[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:12:48 dist-test-slave-f5h6 krb5kdc[10380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:12:48 dist-test-slave-f5h6 krb5kdc[10380](info): set up 2 sockets
Sep 27 15:12:48 dist-test-slave-f5h6 krb5kdc[10380](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 15:12:49.478922 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.010s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:12:49 dist-test-slave-f5h6 krb5kdc[10380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:12:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:12:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:12:49.659225 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:40279
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:40279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:49.995122 10393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:49.995445 10393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:49.995586 10393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:50.003746 10393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:12:50.003916 10393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:50.004004 10393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:50.004078 10393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:12:50.004138 10393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:12:50.014673 10393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:40279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:40279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:50.016074 10393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:50.017508 10393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:50.030607 10401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:50.030849 10393 server_base.cc:1034] running on GCE node
W20260927 15:12:50.030471 10398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:50.030488 10399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:50.031991 10393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:50.033619 10393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:50.034940 10393 hybrid_clock.cc:648] HybridClock initialized: now 1790521970034856 us; error 70 us; skew 500 ppm
I20260927 15:12:50.035359 10393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:50 dist-test-slave-f5h6 krb5kdc[10380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521970, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:50.039028 10393 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:12:50.040927 10393 webserver.cc:460] Webserver started at http://127.10.33.126:38407/ using document root <none> and password file <none>
I20260927 15:12:50.042490 10393 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:50.042660 10393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:50.043009 10393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:50.046581 10393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "99f9ae27b1e84844819b5bfb67a2cc0a"
format_stamp: "Formatted at 2026-09-27 15:12:50 on dist-test-slave-f5h6"
server_key: "978eab4ef994140e6809cc68871a5210"
server_key_iv: "ee8d2f03dd0aa9268881938193fc5b9b"
server_key_version: "encryptionkey@0"
I20260927 15:12:50.047420 10393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "99f9ae27b1e84844819b5bfb67a2cc0a"
format_stamp: "Formatted at 2026-09-27 15:12:50 on dist-test-slave-f5h6"
server_key: "978eab4ef994140e6809cc68871a5210"
server_key_iv: "ee8d2f03dd0aa9268881938193fc5b9b"
server_key_version: "encryptionkey@0"
I20260927 15:12:50.053422 10393 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 15:12:50.058089 10408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:50.060178 10393 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:12:50.060420 10393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "99f9ae27b1e84844819b5bfb67a2cc0a"
format_stamp: "Formatted at 2026-09-27 15:12:50 on dist-test-slave-f5h6"
server_key: "978eab4ef994140e6809cc68871a5210"
server_key_iv: "ee8d2f03dd0aa9268881938193fc5b9b"
server_key_version: "encryptionkey@0"
I20260927 15:12:50.060767 10393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:50.083437 10393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:50.088599 10393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:50.088882 10393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:50.108544 10393 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:40279
I20260927 15:12:50.108564 10459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:40279 every 8 connection(s)
I20260927 15:12:50.110260 10393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:12:50.114809 10460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:50.117048 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 10393
I20260927 15:12:50.117381 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:12:50.119319 10373 external_mini_cluster.cc:1428] Setting key bda48164d3be3e244223e642ad30783a
I20260927 15:12:50.126986 10460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a: Bootstrap starting.
I20260927 15:12:50.130132 10460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:50.131390 10460 log.cc:824] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:50.134583 10460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a: No bootstrap required, opened a new log
I20260927 15:12:50.142537 10460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f9ae27b1e84844819b5bfb67a2cc0a" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40279 } }
I20260927 15:12:50.142957 10460 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:12:50.143055 10460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f9ae27b1e84844819b5bfb67a2cc0a, State: Initialized, Role: FOLLOWER
I20260927 15:12:50.143889 10460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f9ae27b1e84844819b5bfb67a2cc0a" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40279 } }
I20260927 15:12:50.144172 10460 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:12:50.144322 10460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:12:50.144495 10460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:50.146792 10460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f9ae27b1e84844819b5bfb67a2cc0a" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40279 } }
I20260927 15:12:50.147333 10460 leader_election.cc:302] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99f9ae27b1e84844819b5bfb67a2cc0a; no voters:
I20260927 15:12:50.147980 10460 leader_election.cc:288] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:12:50.148109 10465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:12:50.149139 10465 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a [term 1 LEADER]: Becoming Leader. State: Replica: 99f9ae27b1e84844819b5bfb67a2cc0a, State: Running, Role: LEADER
Sep 27 15:12:50 dist-test-slave-f5h6 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:12:50.149736 10465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f9ae27b1e84844819b5bfb67a2cc0a" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40279 } }
I20260927 15:12:50.150403 10460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:12:50.153318 10466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99f9ae27b1e84844819b5bfb67a2cc0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f9ae27b1e84844819b5bfb67a2cc0a" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40279 } } }
I20260927 15:12:50.153825 10466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:50.154330 10467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99f9ae27b1e84844819b5bfb67a2cc0a. Latest consensus state: current_term: 1 leader_uuid: "99f9ae27b1e84844819b5bfb67a2cc0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f9ae27b1e84844819b5bfb67a2cc0a" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40279 } } }
I20260927 15:12:50.154620 10467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:50.157325 10473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:12:50.163631 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:50.123401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48642 (local address 127.10.33.126:40279)
0927 15:12:50.124320 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:50.124343 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:50.125740 (+ 1397us) server_negotiation.cc:407] Connection header received
0927 15:12:50.135663 (+ 9923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:50.135689 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:50.136161 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:50.136609 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:50.140023 (+ 3414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:50.141714 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:50.143738 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:50.144599 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:50.144822 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:50.151158 (+ 6336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:50.151168 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:50.151186 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:50.151235 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:50.156527 (+ 5292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:50.157784 (+ 1257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:50.157795 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:50.157806 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:50.158085 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:50.158759 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:50.158765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:50.158771 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:50.159652 (+ 881us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:50.160035 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:50.161379 (+ 1344us) server_negotiation.cc:299] Negotiation successful
0927 15:12:50.161924 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":413,"threads_started":1}
I20260927 15:12:50.176919 10473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:12:50.187357 10473 catalog_manager.cc:1347] Generated new cluster ID: 3abec93f6c774e29bdd5995582df43e1
I20260927 15:12:50.187486 10473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:12:50.219765 10473 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:12:50.221179 10473 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:12:50.240265 10473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99f9ae27b1e84844819b5bfb67a2cc0a: Generated new TSK 0
I20260927 15:12:50.241506 10473 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.33.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
I20260927 15:12:50.285610 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:40279
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:50.624094 10486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:50.624414 10486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:50.624509 10486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:50.631968 10486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:50.632095 10486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:50.632211 10486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:12:50.642520 10486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:40279
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:50.643823 10486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:50.645174 10486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:50.658838 10494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:50.658893 10491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:50.658835 10492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:50.659598 10486 server_base.cc:1034] running on GCE node
I20260927 15:12:50.660281 10486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:50.661382 10486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:50.662598 10486 hybrid_clock.cc:648] HybridClock initialized: now 1790521970662533 us; error 54 us; skew 500 ppm
I20260927 15:12:50.663007 10486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:50 dist-test-slave-f5h6 krb5kdc[10380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521970, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:50.666544 10486 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:12:50.668107 10486 webserver.cc:460] Webserver started at http://127.10.33.65:40531/ using document root <none> and password file <none>
I20260927 15:12:50.668800 10486 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:50.668923 10486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:50.669183 10486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:50.672178 10486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "504753d17b6a478c9025160d9054fc87"
format_stamp: "Formatted at 2026-09-27 15:12:50 on dist-test-slave-f5h6"
server_key: "106a0b11e24534cf567acebe50d60724"
server_key_iv: "6a6bfb012001ea33899514a12d177449"
server_key_version: "encryptionkey@0"
I20260927 15:12:50.672932 10486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "504753d17b6a478c9025160d9054fc87"
format_stamp: "Formatted at 2026-09-27 15:12:50 on dist-test-slave-f5h6"
server_key: "106a0b11e24534cf567acebe50d60724"
server_key_iv: "6a6bfb012001ea33899514a12d177449"
server_key_version: "encryptionkey@0"
I20260927 15:12:50.678694 10486 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 15:12:50.682685 10501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:50.684309 10486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:12:50.684547 10486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "504753d17b6a478c9025160d9054fc87"
format_stamp: "Formatted at 2026-09-27 15:12:50 on dist-test-slave-f5h6"
server_key: "106a0b11e24534cf567acebe50d60724"
server_key_iv: "6a6bfb012001ea33899514a12d177449"
server_key_version: "encryptionkey@0"
I20260927 15:12:50.684762 10486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:50.710296 10486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:50.715965 10486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:50.716415 10486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:50.717515 10486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:12:50.719043 10486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:12:50.719131 10486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:50.719257 10486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:12:50.719309 10486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:50.752566 10486 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:33333
I20260927 15:12:50.752600 10613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:33333 every 8 connection(s)
I20260927 15:12:50.753983 10486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:12:50.764200 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 10486
I20260927 15:12:50.764458 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:12:50.765312 10373 external_mini_cluster.cc:1428] Setting key 3a40213bc86f1ee57c50e4947afc2d0e
Sep 27 15:12:50 dist-test-slave-f5h6 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521970, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:12:50.776260 10616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:50.756620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:41645 (local address 127.10.33.126:40279)
0927 15:12:50.757047 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:50.757052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:50.758224 (+ 1172us) server_negotiation.cc:407] Connection header received
0927 15:12:50.760230 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:50.760236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:50.760363 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:50.760473 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:50.762200 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:50.762964 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:50.765461 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:50.766147 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:50.766278 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:50.770305 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:50.770312 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:50.770316 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:50.770356 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:50.772986 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:50.773801 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:50.773806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:50.773809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:50.773982 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:50.774439 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:50.774444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:50.774447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:50.774722 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:50.774879 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:50.775637 (+ 758us) server_negotiation.cc:299] Negotiation successful
0927 15:12:50.775900 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":210,"threads_started":1}
I20260927 15:12:50.777318 10617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:50.757227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40279 (local address 127.10.33.65:41645)
0927 15:12:50.758039 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:50.758087 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:50.759883 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:50.760712 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:50.760722 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:50.761300 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:50.762008 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:50.762030 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:50.763142 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:50.763148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:50.765281 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:50.765308 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:50.766452 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:50.766460 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:50.766712 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:50.767462 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:50.767494 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:50.770066 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:50.773161 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:50.773171 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:50.773189 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:50.773662 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:50.774094 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:50.774099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:50.774103 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:50.774305 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:50.774849 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:50.774857 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:50.775221 (+ 364us) client_negotiation.cc:769] Sending connection context
0927 15:12:50.775492 (+ 271us) client_negotiation.cc:240] Negotiation successful
0927 15:12:50.776073 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":270,"threads_started":1}
I20260927 15:12:50.779345 10614 heartbeater.cc:347] Connected to a master server at 127.10.33.126:40279
I20260927 15:12:50.779664 10614 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.33.66@KRBTEST.COM; defaulting to no policy
I20260927 15:12:50.780417 10614 heartbeater.cc:511] Master 127.10.33.126:40279 requested a full tablet report, sending...
I20260927 15:12:50.782815 10425 ts_manager.cc:194] Registered new tserver with Master: 504753d17b6a478c9025160d9054fc87 (127.10.33.65:33333)
I20260927 15:12:50.784551 10425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:41645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
I20260927 15:12:50.819808 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:40279
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:51.145092 10620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:51.145401 10620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:51.145496 10620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:51.153060 10620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:51.153187 10620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:51.153311 10620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:12:51.163563 10620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:40279
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:51.164914 10620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:51.166414 10620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:51.179971 10628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:51.180013 10625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:51.179949 10626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:51.180984 10620 server_base.cc:1034] running on GCE node
I20260927 15:12:51.181648 10620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:51.182727 10620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:51.183951 10620 hybrid_clock.cc:648] HybridClock initialized: now 1790521971183889 us; error 52 us; skew 500 ppm
I20260927 15:12:51.184275 10620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:51 dist-test-slave-f5h6 krb5kdc[10380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521971, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:51.187584 10620 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:12:51.190294 10620 webserver.cc:460] Webserver started at http://127.10.33.66:42779/ using document root <none> and password file <none>
I20260927 15:12:51.191056 10620 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:51.191193 10620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:51.191512 10620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:51.194761 10620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "79e6d8efdb7e4dfea411e62719a8702f"
format_stamp: "Formatted at 2026-09-27 15:12:51 on dist-test-slave-f5h6"
server_key: "3833735af03f1a89b22ce255b4254410"
server_key_iv: "cd5811dfee30ea96fa7696734f67a7d9"
server_key_version: "encryptionkey@0"
I20260927 15:12:51.195595 10620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "79e6d8efdb7e4dfea411e62719a8702f"
format_stamp: "Formatted at 2026-09-27 15:12:51 on dist-test-slave-f5h6"
server_key: "3833735af03f1a89b22ce255b4254410"
server_key_iv: "cd5811dfee30ea96fa7696734f67a7d9"
server_key_version: "encryptionkey@0"
I20260927 15:12:51.201314 10620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:12:51.205662 10635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:51.207319 10620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:12:51.207496 10620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "79e6d8efdb7e4dfea411e62719a8702f"
format_stamp: "Formatted at 2026-09-27 15:12:51 on dist-test-slave-f5h6"
server_key: "3833735af03f1a89b22ce255b4254410"
server_key_iv: "cd5811dfee30ea96fa7696734f67a7d9"
server_key_version: "encryptionkey@0"
I20260927 15:12:51.207706 10620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:51.239022 10620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:51.243661 10620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:51.243963 10620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:51.244966 10620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:12:51.246459 10620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:12:51.246538 10620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:51.246654 10620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:12:51.246708 10620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:51.277412 10620 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:40015
I20260927 15:12:51.277469 10747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:40015 every 8 connection(s)
I20260927 15:12:51.278820 10620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:12:51.287182 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 10620
I20260927 15:12:51.287389 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:12:51.287940 10373 external_mini_cluster.cc:1428] Setting key 12195970da1530a39806c87f9e0f6e3a
Sep 27 15:12:51 dist-test-slave-f5h6 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521971, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:12:51.301671 10750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:51.281344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:39249 (local address 127.10.33.126:40279)
0927 15:12:51.281772 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:51.281777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:51.282764 (+ 987us) server_negotiation.cc:407] Connection header received
0927 15:12:51.284581 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:51.284587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:51.284695 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:51.284791 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:51.286400 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:51.287265 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:51.289564 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:51.290193 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:51.290333 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:51.294748 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:51.294755 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:51.294760 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:51.294801 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:51.297998 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:51.298774 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:51.298779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:51.298783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:51.298936 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:51.299411 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:51.299416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:51.299419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:51.299747 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:51.300020 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:51.301021 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0927 15:12:51.301277 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":209,"threads_started":1}
I20260927 15:12:51.302501 10751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:51.281756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40279 (local address 127.10.33.66:39249)
0927 15:12:51.282575 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:51.282622 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:51.284295 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:51.285047 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:51.285057 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:51.285587 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:51.286230 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:51.286249 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:51.287474 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:51.287479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:51.289403 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:51.289417 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:51.290523 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:51.290532 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:51.290874 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:51.291632 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:51.291662 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:51.294460 (+ 2798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:51.298156 (+ 3696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:51.298165 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:51.298180 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:51.298640 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:51.299054 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:51.299057 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:51.299060 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:51.299256 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:51.300019 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:51.300026 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:51.300584 (+ 558us) client_negotiation.cc:769] Sending connection context
0927 15:12:51.300852 (+ 268us) client_negotiation.cc:240] Negotiation successful
0927 15:12:51.301294 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":349,"threads_started":1}
I20260927 15:12:51.304039 10748 heartbeater.cc:347] Connected to a master server at 127.10.33.126:40279
I20260927 15:12:51.304375 10748 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.33.67@KRBTEST.COM; defaulting to no policy
I20260927 15:12:51.305167 10748 heartbeater.cc:511] Master 127.10.33.126:40279 requested a full tablet report, sending...
I20260927 15:12:51.306782 10425 ts_manager.cc:194] Registered new tserver with Master: 79e6d8efdb7e4dfea411e62719a8702f (127.10.33.66:40015)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.67@KRBTEST.COM" created.
I20260927 15:12:51.307685 10425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:39249
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
I20260927 15:12:51.335405 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:40279
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:51.658484 10754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:51.658785 10754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:51.658922 10754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:51.666493 10754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:51.666622 10754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:51.666767 10754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:12:51.676623 10754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:40279
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:51.677987 10754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:51.679286 10754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:51.692775 10760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:51.692799 10759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:51.692834 10762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:51.693497 10754 server_base.cc:1034] running on GCE node
I20260927 15:12:51.694166 10754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:51.695156 10754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:51.696385 10754 hybrid_clock.cc:648] HybridClock initialized: now 1790521971696310 us; error 68 us; skew 500 ppm
I20260927 15:12:51.696722 10754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:51 dist-test-slave-f5h6 krb5kdc[10380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521971, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:51.700088 10754 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:12:51.701962 10754 webserver.cc:460] Webserver started at http://127.10.33.67:38313/ using document root <none> and password file <none>
I20260927 15:12:51.702706 10754 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:51.702833 10754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:51.703205 10754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:51.706382 10754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "2cfb4b40b2444a23a0f853351d9b91ca"
format_stamp: "Formatted at 2026-09-27 15:12:51 on dist-test-slave-f5h6"
server_key: "2a34f7bfe87820e282dfa49f7beff7c7"
server_key_iv: "15bc90cd6b55f4c46ab8f962bbfba287"
server_key_version: "encryptionkey@0"
I20260927 15:12:51.707150 10754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "2cfb4b40b2444a23a0f853351d9b91ca"
format_stamp: "Formatted at 2026-09-27 15:12:51 on dist-test-slave-f5h6"
server_key: "2a34f7bfe87820e282dfa49f7beff7c7"
server_key_iv: "15bc90cd6b55f4c46ab8f962bbfba287"
server_key_version: "encryptionkey@0"
I20260927 15:12:51.712913 10754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:12:51.717391 10769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:51.719916 10754 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260927 15:12:51.720166 10754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "2cfb4b40b2444a23a0f853351d9b91ca"
format_stamp: "Formatted at 2026-09-27 15:12:51 on dist-test-slave-f5h6"
server_key: "2a34f7bfe87820e282dfa49f7beff7c7"
server_key_iv: "15bc90cd6b55f4c46ab8f962bbfba287"
server_key_version: "encryptionkey@0"
I20260927 15:12:51.720443 10754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:51.754881 10754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:51.759310 10754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:51.759589 10754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:51.760576 10754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:12:51.762051 10754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:12:51.762140 10754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:51.762269 10754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:12:51.762326 10754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:51.788481 10614 heartbeater.cc:503] Master 127.10.33.126:40279 was elected leader, sending a full tablet report...
I20260927 15:12:51.793368 10754 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:35447
I20260927 15:12:51.793417 10881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:35447 every 8 connection(s)
I20260927 15:12:51.794728 10754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:12:51.802116 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 10754
I20260927 15:12:51.802292 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:12:51.802886 10373 external_mini_cluster.cc:1428] Setting key 001edd95c2520ac8a8f58eb551c5dded
Sep 27 15:12:51 dist-test-slave-f5h6 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521971, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:12:51.814604 10750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:51.797294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:38181 (local address 127.10.33.126:40279)
0927 15:12:51.797500 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:51.797505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:51.798717 (+ 1212us) server_negotiation.cc:407] Connection header received
0927 15:12:51.800511 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:51.800516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:51.800613 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:51.800713 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:51.802310 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:51.803381 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:51.805393 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:51.805785 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:51.805941 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:51.809308 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:51.809313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:51.809316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:51.809348 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:51.811844 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:51.812555 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:51.812559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:51.812562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:51.812671 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:51.813030 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:51.813033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:51.813034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:51.813296 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:51.813424 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:51.814192 (+ 768us) server_negotiation.cc:299] Negotiation successful
0927 15:12:51.814363 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 15:12:51.815557 10884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:51.797824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40279 (local address 127.10.33.67:38181)
0927 15:12:51.798550 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:51.798590 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:51.800228 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:51.800932 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:51.800940 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:51.801460 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:51.802118 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:51.802139 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:51.803557 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:51.803563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:51.805265 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:51.805278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:51.806036 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:51.806044 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:51.806343 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:51.807016 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:51.807042 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:51.809090 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:51.811957 (+ 2867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:51.811964 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:51.811978 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:51.812445 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:51.812769 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:51.812772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:51.812774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:51.812944 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:51.813419 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:51.813424 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:51.813759 (+ 335us) client_negotiation.cc:769] Sending connection context
0927 15:12:51.814052 (+ 293us) client_negotiation.cc:240] Negotiation successful
0927 15:12:51.814447 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":280,"threads_started":1}
I20260927 15:12:51.816900 10882 heartbeater.cc:347] Connected to a master server at 127.10.33.126:40279
I20260927 15:12:51.817207 10882 heartbeater.cc:464] Registering TS with master...
I20260927 15:12:51.817993 10882 heartbeater.cc:511] Master 127.10.33.126:40279 requested a full tablet report, sending...
I20260927 15:12:51.819504 10425 ts_manager.cc:194] Registered new tserver with Master: 2cfb4b40b2444a23a0f853351d9b91ca (127.10.33.67:35447)
I20260927 15:12:51.820314 10425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:38181
I20260927 15:12:51.831492 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:12:51 dist-test-slave-f5h6 krb5kdc[10380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521971, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:12:51 dist-test-slave-f5h6 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521971, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.33.65@KRBTEST.COM
I20260927 15:12:51.875299 10889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:51.858640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52920 (local address 127.10.33.65:33333)
0927 15:12:51.859365 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:51.859374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:51.859407 (+ 33us) server_negotiation.cc:407] Connection header received
0927 15:12:51.859557 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:51.859564 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:51.859798 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:51.860080 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:51.861538 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:51.862668 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:51.863991 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:51.864629 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:51.864753 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:51.869053 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:51.869068 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:51.869085 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:51.869130 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:51.872153 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:51.872936 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:51.872944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:51.872946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:51.873087 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:51.873486 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:51.873489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:51.873492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:51.874013 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:51.874316 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:51.874667 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 15:12:51.874946 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":390,"threads_started":1}
W20260927 15:12:51.876685 10529 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52920
Sep 27 15:12:51 dist-test-slave-f5h6 krb5kdc[10380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:12:51 dist-test-slave-f5h6 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.65@KRBTEST.COM
I20260927 15:12:51.919909 10889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:51.905791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.10.33.65:33333)
0927 15:12:51.906162 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:51.906169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:51.906373 (+ 204us) server_negotiation.cc:407] Connection header received
0927 15:12:51.906520 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:51.906527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:51.906595 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:51.906692 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:51.908074 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:51.908810 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:51.910169 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:51.910618 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:51.910764 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:51.914820 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:51.914825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:51.914829 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:51.914870 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:51.917400 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:51.918162 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:51.918167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:51.918169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:51.918313 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:51.918666 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:51.918670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:51.918672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:51.918952 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:51.919130 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:51.919398 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 15:12:51.919580 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260927 15:12:51.921370 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 10486
I20260927 15:12:51.934587 10889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:51.922447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52926 (local address 127.10.33.65:33333)
0927 15:12:51.922646 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:51.922652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:51.922779 (+ 127us) server_negotiation.cc:407] Connection header received
0927 15:12:51.922948 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:51.922952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:51.923010 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:51.923091 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:51.924340 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:51.925104 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:51.926482 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:51.926935 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:51.927041 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:51.929638 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:51.929643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:51.929646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:51.929683 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:51.932252 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:51.932921 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:51.932925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:51.932927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:51.933057 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:51.933341 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:51.933345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:51.933346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:51.933592 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:51.933769 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:51.934041 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 15:12:51.934216 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:12:51.935395 10609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:12:52.020761 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 10486
I20260927 15:12:52.034204 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 10620
Sep 27 15:12:52 dist-test-slave-f5h6 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.66@KRBTEST.COM
I20260927 15:12:52.050733 10896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:52.035127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.10.33.66:40015)
0927 15:12:52.035633 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:52.035640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:52.035663 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:12:52.035722 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:52.035728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:52.035936 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:52.036065 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:52.037279 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:52.038390 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:52.039673 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:52.040225 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:52.040390 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:52.044470 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:52.044484 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:52.044499 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:52.044539 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:52.047299 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:52.048115 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:52.048124 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:52.048126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:52.048276 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:52.048855 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:52.048861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:52.048864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:52.049285 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:52.049533 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:52.050082 (+ 549us) server_negotiation.cc:299] Negotiation successful
0927 15:12:52.050306 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":278,"threads_started":1}
I20260927 15:12:52.052202 10743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:12:52.135118 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 10620
I20260927 15:12:52.153614 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 10754
Sep 27 15:12:52 dist-test-slave-f5h6 krb5kdc[10380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.67@KRBTEST.COM
I20260927 15:12:52.170398 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:52.154689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.10.33.67:35447)
0927 15:12:52.155333 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:52.155340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:52.155369 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:12:52.155430 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:52.155436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:52.155644 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:52.155820 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:52.157041 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:52.158128 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:52.159426 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:52.159987 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:52.160102 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:52.164360 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:52.164377 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:52.164393 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:52.164454 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:52.167488 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:52.168304 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:52.168313 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:52.168315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:52.168461 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:52.168833 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:52.168837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:52.168840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:52.169268 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:52.169522 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:52.169844 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 15:12:52.170106 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":329,"threads_started":1}
I20260927 15:12:52.171703 10877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:12:52.254700 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 10754
I20260927 15:12:52.268031 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 10393
I20260927 15:12:52.269170 10455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:12:52.370362 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 10393
2026-09-27T15:12:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3939 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:12:52 dist-test-slave-f5h6 krb5kdc[10903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:12:52 dist-test-slave-f5h6 krb5kdc[10903](info): set up 2 sockets
Sep 27 15:12:52 dist-test-slave-f5h6 krb5kdc[10903](info): commencing operation
krb5kdc: starting...
W20260927 15:12:53.361840 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.007s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:12:53 dist-test-slave-f5h6 krb5kdc[10903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:12:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:12:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:12:53.550134 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:41661
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:41661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:53.873329 10916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:53.873651 10916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:53.873772 10916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:53.881034 10916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:12:53.881173 10916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:53.881250 10916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:53.881385 10916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:12:53.881446 10916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:12:53.892858 10916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:41661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:41661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:53.894291 10916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:53.895627 10916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:53.913043 10922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:53.913008 10924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:53.913072 10921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:53.914031 10916 server_base.cc:1034] running on GCE node
I20260927 15:12:53.914927 10916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:53.916427 10916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:53.917632 10916 hybrid_clock.cc:648] HybridClock initialized: now 1790521973917581 us; error 38 us; skew 500 ppm
I20260927 15:12:53.918069 10916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:53 dist-test-slave-f5h6 krb5kdc[10903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521973, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:53.927724 10916 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:12:53.929559 10916 webserver.cc:460] Webserver started at http://127.10.33.126:33375/ using document root <none> and password file <none>
I20260927 15:12:53.930366 10916 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:53.930481 10916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:53.930799 10916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:53.934085 10916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "aacf14f3e9dc4b928376fe150c271f76"
format_stamp: "Formatted at 2026-09-27 15:12:53 on dist-test-slave-f5h6"
server_key: "94a0e5ac1de0f3dff4ecb66499e17922"
server_key_iv: "17cb588fac7dbcecfa817f0f73648d8d"
server_key_version: "encryptionkey@0"
I20260927 15:12:53.934891 10916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "aacf14f3e9dc4b928376fe150c271f76"
format_stamp: "Formatted at 2026-09-27 15:12:53 on dist-test-slave-f5h6"
server_key: "94a0e5ac1de0f3dff4ecb66499e17922"
server_key_iv: "17cb588fac7dbcecfa817f0f73648d8d"
server_key_version: "encryptionkey@0"
I20260927 15:12:53.940505 10916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:12:53.944566 10931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:53.946146 10916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:12:53.946389 10916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "aacf14f3e9dc4b928376fe150c271f76"
format_stamp: "Formatted at 2026-09-27 15:12:53 on dist-test-slave-f5h6"
server_key: "94a0e5ac1de0f3dff4ecb66499e17922"
server_key_iv: "17cb588fac7dbcecfa817f0f73648d8d"
server_key_version: "encryptionkey@0"
I20260927 15:12:53.946601 10916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:53.973632 10916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:53.978112 10916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:53.978389 10916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:53.997413 10916 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:41661
I20260927 15:12:53.997571 10982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:41661 every 8 connection(s)
I20260927 15:12:53.999217 10916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:12:54.003408 10983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:54.008131 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 10916
I20260927 15:12:54.008473 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:12:54.009171 10373 external_mini_cluster.cc:1428] Setting key be8acf8637cad9f5dec69c4eb3cb5308
I20260927 15:12:54.016722 10983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76: Bootstrap starting.
I20260927 15:12:54.020527 10983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:54.021726 10983 log.cc:824] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:12:54 dist-test-slave-f5h6 krb5kdc[10903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:12:54.024986 10983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76: No bootstrap required, opened a new log
I20260927 15:12:54.033578 10986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:54.011387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52526 (local address 127.10.33.126:41661)
0927 15:12:54.012219 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:54.012236 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:54.012291 (+ 55us) server_negotiation.cc:407] Connection header received
0927 15:12:54.014830 (+ 2539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:54.014864 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:54.015442 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:54.015925 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:54.017418 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:54.019493 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:54.020883 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:54.021654 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:54.021901 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:54.025923 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:54.025933 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:54.025949 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:54.025984 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:54.029216 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:54.029994 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:54.030001 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:54.030010 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:54.030209 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:54.030602 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:54.030606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:54.030608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:54.031199 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:54.031490 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:54.032051 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 15:12:54.032490 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":347,"threads_started":1}
I20260927 15:12:54.033753 10983 raft_consensus.cc:348] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aacf14f3e9dc4b928376fe150c271f76" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 41661 } }
I20260927 15:12:54.034214 10983 raft_consensus.cc:374] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:12:54.034315 10983 raft_consensus.cc:729] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aacf14f3e9dc4b928376fe150c271f76, State: Initialized, Role: FOLLOWER
I20260927 15:12:54.034935 10983 consensus_queue.cc:260] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aacf14f3e9dc4b928376fe150c271f76" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 41661 } }
I20260927 15:12:54.035171 10983 raft_consensus.cc:388] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:12:54.035290 10983 raft_consensus.cc:482] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:12:54.035454 10983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:54.037844 10983 raft_consensus.cc:504] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aacf14f3e9dc4b928376fe150c271f76" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 41661 } }
I20260927 15:12:54.038364 10983 leader_election.cc:302] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aacf14f3e9dc4b928376fe150c271f76; no voters:
I20260927 15:12:54.038959 10983 leader_election.cc:288] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:12:54.039047 10988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:12:54.040217 10988 raft_consensus.cc:686] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76 [term 1 LEADER]: Becoming Leader. State: Replica: aacf14f3e9dc4b928376fe150c271f76, State: Running, Role: LEADER
I20260927 15:12:54.040722 10988 consensus_queue.cc:237] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aacf14f3e9dc4b928376fe150c271f76" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 41661 } }
I20260927 15:12:54.041062 10983 sys_catalog.cc:564] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:12:54.043499 10990 sys_catalog.cc:455] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aacf14f3e9dc4b928376fe150c271f76. Latest consensus state: current_term: 1 leader_uuid: "aacf14f3e9dc4b928376fe150c271f76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aacf14f3e9dc4b928376fe150c271f76" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 41661 } } }
I20260927 15:12:54.043475 10989 sys_catalog.cc:455] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aacf14f3e9dc4b928376fe150c271f76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aacf14f3e9dc4b928376fe150c271f76" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 41661 } } }
I20260927 15:12:54.043926 10990 sys_catalog.cc:458] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76 [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:54.043941 10989 sys_catalog.cc:458] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76 [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:54.044790 10996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:12:54.053527 10996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:12:54.062086 10996 catalog_manager.cc:1347] Generated new cluster ID: 69659551ddd049039058e5687cd0abaf
I20260927 15:12:54.062213 10996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:12:54.078290 10996 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:12:54.080376 10996 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:12:54.092525 10996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aacf14f3e9dc4b928376fe150c271f76: Generated new TSK 0
I20260927 15:12:54.093421 10996 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.33.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
I20260927 15:12:54.143924 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:41661
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:54.470916 11009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:54.471190 11009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:54.471309 11009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:54.478564 11009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:54.478679 11009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:54.478832 11009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:12:54.488598 11009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:41661
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:54.489959 11009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:54.491268 11009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:54.504037 11014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:54.504200 11015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:54.504118 11017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:54.505044 11009 server_base.cc:1034] running on GCE node
I20260927 15:12:54.505645 11009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:54.506654 11009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:54.507889 11009 hybrid_clock.cc:648] HybridClock initialized: now 1790521974507828 us; error 46 us; skew 500 ppm
I20260927 15:12:54.508191 11009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:54 dist-test-slave-f5h6 krb5kdc[10903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521974, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:54.511394 11009 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:12:54.512928 11009 webserver.cc:460] Webserver started at http://127.10.33.65:41615/ using document root <none> and password file <none>
I20260927 15:12:54.513631 11009 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:54.513760 11009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:54.514113 11009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:54.517199 11009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "c3b07e7e140748fe964ba5e7d4bb0bb1"
format_stamp: "Formatted at 2026-09-27 15:12:54 on dist-test-slave-f5h6"
server_key: "041560b43677e4f5631194cd73ade87a"
server_key_iv: "0ae3ae0b9409249a4cca0c4141dd5daa"
server_key_version: "encryptionkey@0"
I20260927 15:12:54.518090 11009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "c3b07e7e140748fe964ba5e7d4bb0bb1"
format_stamp: "Formatted at 2026-09-27 15:12:54 on dist-test-slave-f5h6"
server_key: "041560b43677e4f5631194cd73ade87a"
server_key_iv: "0ae3ae0b9409249a4cca0c4141dd5daa"
server_key_version: "encryptionkey@0"
I20260927 15:12:54.523543 11009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:12:54.527565 11024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:54.529179 11009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:12:54.529430 11009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "c3b07e7e140748fe964ba5e7d4bb0bb1"
format_stamp: "Formatted at 2026-09-27 15:12:54 on dist-test-slave-f5h6"
server_key: "041560b43677e4f5631194cd73ade87a"
server_key_iv: "0ae3ae0b9409249a4cca0c4141dd5daa"
server_key_version: "encryptionkey@0"
I20260927 15:12:54.529693 11009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:54.576244 11009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:54.581100 11009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:54.581370 11009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:54.582379 11009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:12:54.583832 11009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:12:54.583935 11009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:54.584048 11009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:12:54.584106 11009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:54.614295 11009 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:34315
I20260927 15:12:54.614346 11136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:34315 every 8 connection(s)
I20260927 15:12:54.615592 11009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:12:54.623579 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 11009
I20260927 15:12:54.623790 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:12:54.624554 10373 external_mini_cluster.cc:1428] Setting key 2e3f4a9e1c5dcedf493bbee75987c250
Sep 27 15:12:54 dist-test-slave-f5h6 krb5kdc[10903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521974, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:12:54.636868 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:54.618137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:33585 (local address 127.10.33.126:41661)
0927 15:12:54.618574 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:54.618580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:54.619617 (+ 1037us) server_negotiation.cc:407] Connection header received
0927 15:12:54.621402 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:54.621407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:54.621474 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:54.621569 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:54.623181 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:54.624357 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:54.626129 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:54.626574 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:54.626709 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:54.630354 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:54.630361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:54.630366 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:54.630408 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:54.632758 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:54.633667 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:54.633671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:54.633674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:54.633797 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:54.634539 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:54.634543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:54.634545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:54.634824 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:54.635113 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:54.636179 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0927 15:12:54.636423 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":203,"threads_started":1}
I20260927 15:12:54.637805 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:54.618735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:41661 (local address 127.10.33.65:33585)
0927 15:12:54.619468 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:54.619510 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:54.621140 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:54.621762 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:54.621771 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:54.622318 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:54.622932 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:54.622950 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:54.624509 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:54.624513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:54.626002 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:54.626015 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:54.626808 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:54.626813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:54.627028 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:54.627721 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:54.627746 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:54.630108 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:54.632936 (+ 2828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:54.632945 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:54.632965 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:54.633533 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:54.634128 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:54.634132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:54.634136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:54.634393 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:54.635131 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:54.635139 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:54.635646 (+ 507us) client_negotiation.cc:769] Sending connection context
0927 15:12:54.636018 (+ 372us) client_negotiation.cc:240] Negotiation successful
0927 15:12:54.636605 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":255,"threads_started":1}
I20260927 15:12:54.639555 11137 heartbeater.cc:347] Connected to a master server at 127.10.33.126:41661
I20260927 15:12:54.639844 11137 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.33.66@KRBTEST.COM; defaulting to no policy
I20260927 15:12:54.640559 11137 heartbeater.cc:511] Master 127.10.33.126:41661 requested a full tablet report, sending...
I20260927 15:12:54.642796 10948 ts_manager.cc:194] Registered new tserver with Master: c3b07e7e140748fe964ba5e7d4bb0bb1 (127.10.33.65:34315)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.66@KRBTEST.COM" created.
I20260927 15:12:54.644375 10948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:33585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
I20260927 15:12:54.666185 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:41661
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:54.982753 11143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:54.983043 11143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:54.983167 11143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:54.990444 11143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:54.990556 11143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:54.990705 11143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:12:55.001199 11143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:41661
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:55.002568 11143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:55.003892 11143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:55.016798 11151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:55.016865 11148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:55.016902 11149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:55.017525 11143 server_base.cc:1034] running on GCE node
I20260927 15:12:55.018185 11143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:55.019188 11143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:55.020388 11143 hybrid_clock.cc:648] HybridClock initialized: now 1790521975020318 us; error 64 us; skew 500 ppm
I20260927 15:12:55.020697 11143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:55 dist-test-slave-f5h6 krb5kdc[10903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521975, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:55.024040 11143 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:12:55.025591 11143 webserver.cc:460] Webserver started at http://127.10.33.66:44869/ using document root <none> and password file <none>
I20260927 15:12:55.026312 11143 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:55.026422 11143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:55.026727 11143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:55.029668 11143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "061711b15c764a5483340f22a6bc38d0"
format_stamp: "Formatted at 2026-09-27 15:12:55 on dist-test-slave-f5h6"
server_key: "e9442a5869126cb3f609bb435c457b8e"
server_key_iv: "85ffc69330e307a590b02d21f1648ebf"
server_key_version: "encryptionkey@0"
I20260927 15:12:55.030534 11143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "061711b15c764a5483340f22a6bc38d0"
format_stamp: "Formatted at 2026-09-27 15:12:55 on dist-test-slave-f5h6"
server_key: "e9442a5869126cb3f609bb435c457b8e"
server_key_iv: "85ffc69330e307a590b02d21f1648ebf"
server_key_version: "encryptionkey@0"
I20260927 15:12:55.035812 11143 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:12:55.039481 11158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:55.040969 11143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:12:55.041126 11143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "061711b15c764a5483340f22a6bc38d0"
format_stamp: "Formatted at 2026-09-27 15:12:55 on dist-test-slave-f5h6"
server_key: "e9442a5869126cb3f609bb435c457b8e"
server_key_iv: "85ffc69330e307a590b02d21f1648ebf"
server_key_version: "encryptionkey@0"
I20260927 15:12:55.041360 11143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:55.062739 11143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:55.067020 11143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:55.067286 11143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:55.068215 11143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:12:55.069554 11143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:12:55.069628 11143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:55.069722 11143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:12:55.069780 11143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:55.101629 11143 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:40893
I20260927 15:12:55.101680 11270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:40893 every 8 connection(s)
I20260927 15:12:55.102872 11143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:12:55.112564 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 11143
I20260927 15:12:55.112746 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:12:55.113312 10373 external_mini_cluster.cc:1428] Setting key c36e007243384699dc239169766f51a4
Sep 27 15:12:55 dist-test-slave-f5h6 krb5kdc[10903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521975, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:12:55.124667 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:55.105448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:39531 (local address 127.10.33.126:41661)
0927 15:12:55.105641 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:55.105645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:55.106875 (+ 1230us) server_negotiation.cc:407] Connection header received
0927 15:12:55.108743 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:55.108748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:55.108822 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:55.108916 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:55.110591 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.111337 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.112915 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.113539 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.113675 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:55.118026 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:55.118032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:55.118036 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:55.118072 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:55.121128 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:55.122092 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:55.122098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:55.122101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:55.122254 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:55.122797 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:55.122802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:55.122805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:55.123133 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:55.123272 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:55.124091 (+ 819us) server_negotiation.cc:299] Negotiation successful
0927 15:12:55.124314 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 15:12:55.125612 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:55.105995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:41661 (local address 127.10.33.66:39531)
0927 15:12:55.106704 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:55.106748 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:55.108414 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:55.109153 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:55.109162 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:55.109709 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:55.110366 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:55.110415 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.111501 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.111505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:55.112763 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:55.112777 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.113761 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.113766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:55.114099 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:55.114919 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:55.114943 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:55.117692 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:55.121387 (+ 3695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:55.121396 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:55.121412 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:55.121954 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:55.122414 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:55.122419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:55.122421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:55.122672 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:55.123323 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:55.123330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:55.123709 (+ 379us) client_negotiation.cc:769] Sending connection context
0927 15:12:55.123972 (+ 263us) client_negotiation.cc:240] Negotiation successful
0927 15:12:55.124406 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":258,"threads_started":1}
I20260927 15:12:55.127055 11271 heartbeater.cc:347] Connected to a master server at 127.10.33.126:41661
I20260927 15:12:55.127353 11271 heartbeater.cc:464] Registering TS with master...
I20260927 15:12:55.128059 11271 heartbeater.cc:511] Master 127.10.33.126:41661 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.33.67@KRBTEST.COM; defaulting to no policy
I20260927 15:12:55.129590 10948 ts_manager.cc:194] Registered new tserver with Master: 061711b15c764a5483340f22a6bc38d0 (127.10.33.66:40893)
I20260927 15:12:55.130529 10948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:39531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
I20260927 15:12:55.153849 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:41661
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:55.472702 11276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:55.473018 11276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:55.473111 11276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:55.480574 11276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:55.480718 11276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:55.481073 11276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:12:55.491739 11276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:41661
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:55.493003 11276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:55.494380 11276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:55.507200 11281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:55.507229 11284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:55.507280 11282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:55.507921 11276 server_base.cc:1034] running on GCE node
I20260927 15:12:55.508551 11276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:55.509542 11276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:55.510774 11276 hybrid_clock.cc:648] HybridClock initialized: now 1790521975510702 us; error 67 us; skew 500 ppm
I20260927 15:12:55.511130 11276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:55 dist-test-slave-f5h6 krb5kdc[10903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521975, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:55.514442 11276 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:12:55.516027 11276 webserver.cc:460] Webserver started at http://127.10.33.67:33385/ using document root <none> and password file <none>
I20260927 15:12:55.516721 11276 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:55.516824 11276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:55.517089 11276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:55.520162 11276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "e90dbffd44634998a77d3938940d7771"
format_stamp: "Formatted at 2026-09-27 15:12:55 on dist-test-slave-f5h6"
server_key: "108ed8d0e0ca41310abebd19634b6677"
server_key_iv: "70fd762ce9e3625b4e29d7de0f692a97"
server_key_version: "encryptionkey@0"
I20260927 15:12:55.520903 11276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "e90dbffd44634998a77d3938940d7771"
format_stamp: "Formatted at 2026-09-27 15:12:55 on dist-test-slave-f5h6"
server_key: "108ed8d0e0ca41310abebd19634b6677"
server_key_iv: "70fd762ce9e3625b4e29d7de0f692a97"
server_key_version: "encryptionkey@0"
I20260927 15:12:55.526115 11276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 15:12:55.529794 11291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:55.531334 11276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:12:55.531529 11276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "e90dbffd44634998a77d3938940d7771"
format_stamp: "Formatted at 2026-09-27 15:12:55 on dist-test-slave-f5h6"
server_key: "108ed8d0e0ca41310abebd19634b6677"
server_key_iv: "70fd762ce9e3625b4e29d7de0f692a97"
server_key_version: "encryptionkey@0"
I20260927 15:12:55.531704 11276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:55.547152 11276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:55.551788 11276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:55.552054 11276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:55.553028 11276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:12:55.554335 11276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:12:55.554406 11276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:55.554497 11276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:12:55.554539 11276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:55.584188 11276 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:45909
I20260927 15:12:55.584226 11403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:45909 every 8 connection(s)
I20260927 15:12:55.585457 11276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:12:55.590329 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 11276
I20260927 15:12:55.590528 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:12:55.591055 10373 external_mini_cluster.cc:1428] Setting key 3aa4f2facae06b1b2094973349614c5d
Sep 27 15:12:55 dist-test-slave-f5h6 krb5kdc[10903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521975, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:12:55.605998 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:55.588214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:43807 (local address 127.10.33.126:41661)
0927 15:12:55.588414 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:55.588419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:55.589643 (+ 1224us) server_negotiation.cc:407] Connection header received
0927 15:12:55.591943 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:55.591949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:55.592066 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:55.592168 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:55.593800 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.594604 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.596207 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.596638 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.596752 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:55.600364 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:55.600369 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:55.600373 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:55.600407 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:55.602962 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:55.603700 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:55.603704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:55.603707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:55.603826 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:55.604241 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:55.604246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:55.604249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:55.604559 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:55.604693 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:55.605414 (+ 721us) server_negotiation.cc:299] Negotiation successful
0927 15:12:55.605627 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 15:12:55.607061 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:55.588750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:41661 (local address 127.10.33.67:43807)
0927 15:12:55.589472 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:55.589515 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:55.591683 (+ 2168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:55.592458 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:55.592466 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:55.593017 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:55.593625 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:55.593645 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.594766 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.594770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:55.596047 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:55.596060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.596880 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.596888 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:55.597136 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:55.597883 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:55.597921 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:55.600111 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:55.603116 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:55.603125 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:55.603144 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:55.603583 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:55.603945 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:55.603950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:55.603952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:55.604134 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:55.604686 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:55.604695 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:55.605052 (+ 357us) client_negotiation.cc:769] Sending connection context
0927 15:12:55.605296 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 15:12:55.605848 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":263,"threads_started":1}
I20260927 15:12:55.608377 11404 heartbeater.cc:347] Connected to a master server at 127.10.33.126:41661
I20260927 15:12:55.608721 11404 heartbeater.cc:464] Registering TS with master...
I20260927 15:12:55.609467 11404 heartbeater.cc:511] Master 127.10.33.126:41661 requested a full tablet report, sending...
I20260927 15:12:55.611057 10948 ts_manager.cc:194] Registered new tserver with Master: e90dbffd44634998a77d3938940d7771 (127.10.33.67:45909)
I20260927 15:12:55.611900 10948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:43807
I20260927 15:12:55.619194 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:12:55.639097 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:55.625907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52540 (local address 127.10.33.126:41661)
0927 15:12:55.626138 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:55.626146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:55.626746 (+ 600us) server_negotiation.cc:407] Connection header received
0927 15:12:55.626908 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:55.626912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:55.626983 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:55.627072 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:55.628452 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.629197 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.630598 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.631111 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.631227 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:55.633650 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:55.633655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:55.633659 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:55.633694 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:55.636382 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:55.637045 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:55.637049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:55.637051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:55.637183 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:55.637671 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:55.637676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:55.637678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:55.638048 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:55.638240 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:55.638592 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 15:12:55.638763 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:12:55.648012 11137 heartbeater.cc:503] Master 127.10.33.126:41661 was elected leader, sending a full tablet report...
I20260927 15:12:55.646191 10948 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:12:55.653595 10948 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:12:55.690182 11421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:55.667273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35118 (local address 127.10.33.67:45909)
0927 15:12:55.682515 (+ 15242us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:55.682525 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:55.683198 (+ 673us) server_negotiation.cc:407] Connection header received
0927 15:12:55.683337 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:55.683343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:55.683528 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:55.683722 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:12:55.684622 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.685658 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.687634 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.689033 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.689151 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:55.689386 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:55.689671 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 15:12:55.689838 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15045,"thread_start_us":15253,"threads_started":1}
I20260927 15:12:55.690258 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:55.666775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:45909 (local address 127.0.0.1:35118)
0927 15:12:55.683077 (+ 16302us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:55.683092 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:55.683200 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:55.683991 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:55.683999 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:55.684024 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:12:55.684424 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:55.684436 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.685795 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.685800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:55.687438 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:55.687454 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.689217 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.689223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:55.689445 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:55.689494 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 15:12:55.689606 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 15:12:55.689800 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16200,"mutex_wait_us":54,"thread_start_us":13435,"threads_started":1}
I20260927 15:12:55.691071 11420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:55.666090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:40893 (local address 127.0.0.1:45794)
0927 15:12:55.679902 (+ 13812us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:55.679932 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:55.680107 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:55.682123 (+ 2016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:55.682132 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:55.682178 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:12:55.682561 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:55.682574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.684816 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.684821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:55.687473 (+ 2652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:55.687492 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.690226 (+ 2734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.690232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:55.690429 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:55.690460 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 15:12:55.690539 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 15:12:55.690666 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13610,"mutex_wait_us":3,"thread_start_us":16750,"threads_started":1}
I20260927 15:12:55.691661 11419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:55.666551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45794 (local address 127.10.33.66:40893)
0927 15:12:55.681035 (+ 14484us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:55.681045 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:55.681080 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:12:55.681194 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:55.681203 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:55.681519 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:55.681741 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:12:55.682823 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.684628 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.687724 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.689304 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.689444 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:55.689660 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:55.691182 (+ 1522us) server_negotiation.cc:299] Negotiation successful
0927 15:12:55.691354 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14334,"thread_start_us":14222,"threads_started":1}
I20260927 15:12:55.694468 11339 tablet_service.cc:1467] Processing CreateTablet for tablet 520bd3a6bff24e48bcc8760ff60b5714 (DEFAULT_TABLE table=test-table [id=fceafd9cd6204afe88fd10848b9da59f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:12:55.696273 11339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 520bd3a6bff24e48bcc8760ff60b5714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:55.697012 11416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:55.665591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:34315 (local address 127.0.0.1:55160)
0927 15:12:55.666426 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:55.666469 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:55.666653 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:55.690858 (+ 24205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:55.690862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:55.690891 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:12:55.691235 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:55.691246 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.692827 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.692832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:55.694308 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:55.694322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.696244 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.696251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:55.696421 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:55.696453 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 15:12:55.696552 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 15:12:55.696686 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":397,"threads_started":1}
I20260927 15:12:55.697093 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:55.665827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55160 (local address 127.10.33.65:34315)
0927 15:12:55.690192 (+ 24365us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:55.690201 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:55.690228 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:12:55.690327 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:55.690336 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:55.690615 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:55.690875 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:12:55.691412 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.692634 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.694490 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.696019 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.696175 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:55.696434 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:55.696609 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 15:12:55.696771 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24193,"thread_start_us":24009,"threads_started":1}
I20260927 15:12:55.697299 11206 tablet_service.cc:1467] Processing CreateTablet for tablet 520bd3a6bff24e48bcc8760ff60b5714 (DEFAULT_TABLE table=test-table [id=fceafd9cd6204afe88fd10848b9da59f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:12:55.698747 11206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 520bd3a6bff24e48bcc8760ff60b5714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:55.700981 11072 tablet_service.cc:1467] Processing CreateTablet for tablet 520bd3a6bff24e48bcc8760ff60b5714 (DEFAULT_TABLE table=test-table [id=fceafd9cd6204afe88fd10848b9da59f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:12:55.702522 11072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 520bd3a6bff24e48bcc8760ff60b5714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:55.709717 11422 tablet_bootstrap.cc:492] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0: Bootstrap starting.
I20260927 15:12:55.713841 11422 tablet_bootstrap.cc:654] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:55.715044 11422 log.cc:824] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:55.717099 11423 tablet_bootstrap.cc:492] T 520bd3a6bff24e48bcc8760ff60b5714 P e90dbffd44634998a77d3938940d7771: Bootstrap starting.
I20260927 15:12:55.718014 11422 tablet_bootstrap.cc:492] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0: No bootstrap required, opened a new log
I20260927 15:12:55.718325 11425 tablet_bootstrap.cc:492] T 520bd3a6bff24e48bcc8760ff60b5714 P c3b07e7e140748fe964ba5e7d4bb0bb1: Bootstrap starting.
I20260927 15:12:55.718425 11422 ts_tablet_manager.cc:1397] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 15:12:55.722251 11423 tablet_bootstrap.cc:654] T 520bd3a6bff24e48bcc8760ff60b5714 P e90dbffd44634998a77d3938940d7771: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:55.722313 11425 tablet_bootstrap.cc:654] T 520bd3a6bff24e48bcc8760ff60b5714 P c3b07e7e140748fe964ba5e7d4bb0bb1: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:55.723325 11423 log.cc:824] T 520bd3a6bff24e48bcc8760ff60b5714 P e90dbffd44634998a77d3938940d7771: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:55.723511 11425 log.cc:824] T 520bd3a6bff24e48bcc8760ff60b5714 P c3b07e7e140748fe964ba5e7d4bb0bb1: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:55.726289 11425 tablet_bootstrap.cc:492] T 520bd3a6bff24e48bcc8760ff60b5714 P c3b07e7e140748fe964ba5e7d4bb0bb1: No bootstrap required, opened a new log
I20260927 15:12:55.726478 11423 tablet_bootstrap.cc:492] T 520bd3a6bff24e48bcc8760ff60b5714 P e90dbffd44634998a77d3938940d7771: No bootstrap required, opened a new log
I20260927 15:12:55.726562 11425 ts_tablet_manager.cc:1397] T 520bd3a6bff24e48bcc8760ff60b5714 P c3b07e7e140748fe964ba5e7d4bb0bb1: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260927 15:12:55.726742 11423 ts_tablet_manager.cc:1397] T 520bd3a6bff24e48bcc8760ff60b5714 P e90dbffd44634998a77d3938940d7771: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 15:12:55.728402 11422 raft_consensus.cc:348] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b07e7e140748fe964ba5e7d4bb0bb1" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 34315 } } peers { permanent_uuid: "e90dbffd44634998a77d3938940d7771" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 45909 } } peers { permanent_uuid: "061711b15c764a5483340f22a6bc38d0" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 40893 } }
I20260927 15:12:55.728865 11422 raft_consensus.cc:374] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:12:55.728993 11422 raft_consensus.cc:729] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 061711b15c764a5483340f22a6bc38d0, State: Initialized, Role: FOLLOWER
I20260927 15:12:55.729667 11422 consensus_queue.cc:260] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b07e7e140748fe964ba5e7d4bb0bb1" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 34315 } } peers { permanent_uuid: "e90dbffd44634998a77d3938940d7771" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 45909 } } peers { permanent_uuid: "061711b15c764a5483340f22a6bc38d0" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 40893 } }
I20260927 15:12:55.730859 11271 heartbeater.cc:503] Master 127.10.33.126:41661 was elected leader, sending a full tablet report...
I20260927 15:12:55.731539 11422 ts_tablet_manager.cc:1428] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 15:12:55.735271 11425 raft_consensus.cc:348] T 520bd3a6bff24e48bcc8760ff60b5714 P c3b07e7e140748fe964ba5e7d4bb0bb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b07e7e140748fe964ba5e7d4bb0bb1" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 34315 } } peers { permanent_uuid: "e90dbffd44634998a77d3938940d7771" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 45909 } } peers { permanent_uuid: "061711b15c764a5483340f22a6bc38d0" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 40893 } }
I20260927 15:12:55.735755 11425 raft_consensus.cc:374] T 520bd3a6bff24e48bcc8760ff60b5714 P c3b07e7e140748fe964ba5e7d4bb0bb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:12:55.735867 11425 raft_consensus.cc:729] T 520bd3a6bff24e48bcc8760ff60b5714 P c3b07e7e140748fe964ba5e7d4bb0bb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b07e7e140748fe964ba5e7d4bb0bb1, State: Initialized, Role: FOLLOWER
I20260927 15:12:55.736126 11423 raft_consensus.cc:348] T 520bd3a6bff24e48bcc8760ff60b5714 P e90dbffd44634998a77d3938940d7771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b07e7e140748fe964ba5e7d4bb0bb1" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 34315 } } peers { permanent_uuid: "e90dbffd44634998a77d3938940d7771" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 45909 } } peers { permanent_uuid: "061711b15c764a5483340f22a6bc38d0" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 40893 } }
I20260927 15:12:55.736619 11423 raft_consensus.cc:374] T 520bd3a6bff24e48bcc8760ff60b5714 P e90dbffd44634998a77d3938940d7771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:12:55.736533 11425 consensus_queue.cc:260] T 520bd3a6bff24e48bcc8760ff60b5714 P c3b07e7e140748fe964ba5e7d4bb0bb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b07e7e140748fe964ba5e7d4bb0bb1" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 34315 } } peers { permanent_uuid: "e90dbffd44634998a77d3938940d7771" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 45909 } } peers { permanent_uuid: "061711b15c764a5483340f22a6bc38d0" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 40893 } }
I20260927 15:12:55.736720 11423 raft_consensus.cc:729] T 520bd3a6bff24e48bcc8760ff60b5714 P e90dbffd44634998a77d3938940d7771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90dbffd44634998a77d3938940d7771, State: Initialized, Role: FOLLOWER
I20260927 15:12:55.737455 11423 consensus_queue.cc:260] T 520bd3a6bff24e48bcc8760ff60b5714 P e90dbffd44634998a77d3938940d7771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b07e7e140748fe964ba5e7d4bb0bb1" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 34315 } } peers { permanent_uuid: "e90dbffd44634998a77d3938940d7771" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 45909 } } peers { permanent_uuid: "061711b15c764a5483340f22a6bc38d0" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 40893 } }
I20260927 15:12:55.738301 11425 ts_tablet_manager.cc:1428] T 520bd3a6bff24e48bcc8760ff60b5714 P c3b07e7e140748fe964ba5e7d4bb0bb1: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260927 15:12:55.738585 11404 heartbeater.cc:503] Master 127.10.33.126:41661 was elected leader, sending a full tablet report...
I20260927 15:12:55.740216 11423 ts_tablet_manager.cc:1428] T 520bd3a6bff24e48bcc8760ff60b5714 P e90dbffd44634998a77d3938940d7771: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260927 15:12:55.837083 11405 tablet.cc:2369] T 520bd3a6bff24e48bcc8760ff60b5714 P e90dbffd44634998a77d3938940d7771: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:12:55.854897 11272 tablet.cc:2369] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:12:55.868093 11138 tablet.cc:2369] T 520bd3a6bff24e48bcc8760ff60b5714 P c3b07e7e140748fe964ba5e7d4bb0bb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:12:55.894589 11428 raft_consensus.cc:482] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:12:55.894891 11428 raft_consensus.cc:504] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b07e7e140748fe964ba5e7d4bb0bb1" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 34315 } } peers { permanent_uuid: "e90dbffd44634998a77d3938940d7771" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 45909 } } peers { permanent_uuid: "061711b15c764a5483340f22a6bc38d0" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 40893 } }
I20260927 15:12:55.896579 11428 leader_election.cc:288] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b07e7e140748fe964ba5e7d4bb0bb1 (127.10.33.65:34315), e90dbffd44634998a77d3938940d7771 (127.10.33.67:45909)
I20260927 15:12:55.903971 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:55.897046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:46227 (local address 127.10.33.65:34315)
0927 15:12:55.897335 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:55.897342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:55.897688 (+ 346us) server_negotiation.cc:407] Connection header received
0927 15:12:55.898021 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:55.898027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:55.898105 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:55.898171 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:12:55.899022 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.900157 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.902042 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.902967 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.903088 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:55.903142 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:55.903577 (+ 435us) server_negotiation.cc:299] Negotiation successful
0927 15:12:55.903716 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:12:55.903992 11431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:55.896900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:34315 (local address 127.10.33.66:46227)
0927 15:12:55.897569 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:55.897593 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:55.897818 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:55.898390 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:55.898394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:55.898430 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:12:55.898813 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:55.898825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.900294 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.900300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:55.901891 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:55.901903 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.903187 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.903194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:55.903369 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:55.903406 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 15:12:55.903516 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 15:12:55.903660 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":345,"threads_started":1}
I20260927 15:12:55.904459 11432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:55.897765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:45909 (local address 127.10.33.66:33117)
0927 15:12:55.898395 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:55.898413 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:55.898637 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:55.899026 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:55.899030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:55.899052 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:12:55.899378 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:55.899389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.900572 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.900577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:55.902337 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:55.902351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.903673 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.903680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:55.903899 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:55.903931 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 15:12:55.904045 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 15:12:55.904166 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":332,"threads_started":1}
I20260927 15:12:55.904505 11421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:55.897911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:33117 (local address 127.10.33.67:45909)
0927 15:12:55.898093 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:55.898100 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:55.898507 (+ 407us) server_negotiation.cc:407] Connection header received
0927 15:12:55.898824 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:55.898831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:55.898926 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:55.899001 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:12:55.899522 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.900416 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.902506 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.903467 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.903611 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:55.903675 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:55.904089 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 15:12:55.904215 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:12:55.905071 11092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520bd3a6bff24e48bcc8760ff60b5714" candidate_uuid: "061711b15c764a5483340f22a6bc38d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b07e7e140748fe964ba5e7d4bb0bb1" is_pre_election: true
I20260927 15:12:55.905325 11359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520bd3a6bff24e48bcc8760ff60b5714" candidate_uuid: "061711b15c764a5483340f22a6bc38d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e90dbffd44634998a77d3938940d7771" is_pre_election: true
I20260927 15:12:55.905576 11092 raft_consensus.cc:2454] T 520bd3a6bff24e48bcc8760ff60b5714 P c3b07e7e140748fe964ba5e7d4bb0bb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 061711b15c764a5483340f22a6bc38d0 in term 0.
I20260927 15:12:55.905802 11359 raft_consensus.cc:2454] T 520bd3a6bff24e48bcc8760ff60b5714 P e90dbffd44634998a77d3938940d7771 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 061711b15c764a5483340f22a6bc38d0 in term 0.
I20260927 15:12:55.906219 11161 leader_election.cc:302] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 061711b15c764a5483340f22a6bc38d0, c3b07e7e140748fe964ba5e7d4bb0bb1; no voters:
I20260927 15:12:55.906623 11428 raft_consensus.cc:2781] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:12:55.906785 11428 raft_consensus.cc:482] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:12:55.906884 11428 raft_consensus.cc:3037] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:55.909016 11428 raft_consensus.cc:504] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b07e7e140748fe964ba5e7d4bb0bb1" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 34315 } } peers { permanent_uuid: "e90dbffd44634998a77d3938940d7771" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 45909 } } peers { permanent_uuid: "061711b15c764a5483340f22a6bc38d0" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 40893 } }
I20260927 15:12:55.909554 11428 leader_election.cc:288] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0 [CANDIDATE]: Term 1 election: Requested vote from peers c3b07e7e140748fe964ba5e7d4bb0bb1 (127.10.33.65:34315), e90dbffd44634998a77d3938940d7771 (127.10.33.67:45909)
I20260927 15:12:55.910064 11092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520bd3a6bff24e48bcc8760ff60b5714" candidate_uuid: "061711b15c764a5483340f22a6bc38d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b07e7e140748fe964ba5e7d4bb0bb1"
I20260927 15:12:55.910120 11359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520bd3a6bff24e48bcc8760ff60b5714" candidate_uuid: "061711b15c764a5483340f22a6bc38d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e90dbffd44634998a77d3938940d7771"
I20260927 15:12:55.910252 11092 raft_consensus.cc:3037] T 520bd3a6bff24e48bcc8760ff60b5714 P c3b07e7e140748fe964ba5e7d4bb0bb1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:55.910301 11359 raft_consensus.cc:3037] T 520bd3a6bff24e48bcc8760ff60b5714 P e90dbffd44634998a77d3938940d7771 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:55.912395 11359 raft_consensus.cc:2454] T 520bd3a6bff24e48bcc8760ff60b5714 P e90dbffd44634998a77d3938940d7771 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 061711b15c764a5483340f22a6bc38d0 in term 1.
I20260927 15:12:55.912436 11092 raft_consensus.cc:2454] T 520bd3a6bff24e48bcc8760ff60b5714 P c3b07e7e140748fe964ba5e7d4bb0bb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 061711b15c764a5483340f22a6bc38d0 in term 1.
I20260927 15:12:55.912772 11161 leader_election.cc:302] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 061711b15c764a5483340f22a6bc38d0, e90dbffd44634998a77d3938940d7771; no voters:
I20260927 15:12:55.913043 11428 raft_consensus.cc:2781] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:12:55.913520 11428 raft_consensus.cc:686] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0 [term 1 LEADER]: Becoming Leader. State: Replica: 061711b15c764a5483340f22a6bc38d0, State: Running, Role: LEADER
I20260927 15:12:55.914109 11428 consensus_queue.cc:237] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b07e7e140748fe964ba5e7d4bb0bb1" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 34315 } } peers { permanent_uuid: "e90dbffd44634998a77d3938940d7771" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 45909 } } peers { permanent_uuid: "061711b15c764a5483340f22a6bc38d0" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 40893 } }
I20260927 15:12:55.918736 10948 catalog_manager.cc:5495] T 520bd3a6bff24e48bcc8760ff60b5714 P 061711b15c764a5483340f22a6bc38d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 061711b15c764a5483340f22a6bc38d0 (127.10.33.66). New cstate: current_term: 1 leader_uuid: "061711b15c764a5483340f22a6bc38d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b07e7e140748fe964ba5e7d4bb0bb1" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 34315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e90dbffd44634998a77d3938940d7771" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 45909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "061711b15c764a5483340f22a6bc38d0" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 40893 } health_report { overall_health: HEALTHY } } }
Sep 27 15:12:55 dist-test-slave-f5h6 krb5kdc[10903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.65@KRBTEST.COM
I20260927 15:12:55.996976 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:55.980633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55164 (local address 127.10.33.65:34315)
0927 15:12:55.980856 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:55.980863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:55.981359 (+ 496us) server_negotiation.cc:407] Connection header received
0927 15:12:55.981485 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:55.981489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:55.981552 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:55.981647 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:55.983188 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.983956 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.985234 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:55.985660 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:55.985771 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:55.990284 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:55.990303 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:55.990318 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:55.990361 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:55.993772 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:55.994694 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:55.994705 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:55.994709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:55.994871 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:55.995281 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:55.995284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:55.995287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:55.995679 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:55.995907 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:55.996408 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 15:12:55.996594 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 15:12:56 dist-test-slave-f5h6 krb5kdc[10903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521976, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:12:56 dist-test-slave-f5h6 krb5kdc[10903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521976, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.33.65@KRBTEST.COM
I20260927 15:12:56.046326 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:56.031566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55176 (local address 127.10.33.65:34315)
0927 15:12:56.031831 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:56.031837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:56.032022 (+ 185us) server_negotiation.cc:407] Connection header received
0927 15:12:56.032162 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:56.032166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:56.032229 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:56.032311 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:56.033839 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:56.034715 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:56.036075 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:56.036518 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:56.036616 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:56.040806 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:56.040812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:56.040816 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:56.040858 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:56.043513 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:56.044234 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:56.044239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:56.044241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:56.044405 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:56.044941 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:56.044950 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:56.044953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:56.045251 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:56.045516 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:56.045741 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 15:12:56.046011 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
Sep 27 15:12:56 dist-test-slave-f5h6 krb5kdc[10903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:12:56 dist-test-slave-f5h6 krb5kdc[10903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.65@KRBTEST.COM
I20260927 15:12:56.092800 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:56.078564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55184 (local address 127.10.33.65:34315)
0927 15:12:56.078759 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:56.078766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:56.079076 (+ 310us) server_negotiation.cc:407] Connection header received
0927 15:12:56.079247 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:56.079252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:56.079324 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:56.079413 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:56.080800 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:56.081500 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:56.082774 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:56.083203 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:56.083314 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:56.087397 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:56.087403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:56.087406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:56.087448 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:56.090067 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:56.090762 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:56.090767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:56.090769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:56.090900 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:56.091258 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:56.091262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:56.091264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:56.091573 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:56.091827 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:56.092099 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 15:12:56.092435 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 15:12:56.094676 11052 tablet_service.cc:2908] Rejecting scan request for tablet 520bd3a6bff24e48bcc8760ff60b5714: Uninitialized: clean time has not yet been initialized
I20260927 15:12:56.098284 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 11009
I20260927 15:12:56.112578 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:56.099296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.10.33.65:34315)
0927 15:12:56.099655 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:56.099660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:56.099826 (+ 166us) server_negotiation.cc:407] Connection header received
0927 15:12:56.099967 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:56.099971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:56.100042 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:56.100131 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:56.101321 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:56.102214 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:56.103377 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:56.103786 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:56.103919 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:56.106269 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:56.106274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:56.106277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:56.106309 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:56.109152 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:56.109962 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:56.109967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:56.109969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:56.110121 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:56.110571 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:56.110577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:56.110581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:56.110947 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:56.111292 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:56.111991 (+ 699us) server_negotiation.cc:299] Negotiation successful
0927 15:12:56.112242 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235}
I20260927 15:12:56.113550 11132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:12:56.204886 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 11009
I20260927 15:12:56.219024 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 11143
Sep 27 15:12:56 dist-test-slave-f5h6 krb5kdc[10903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.66@KRBTEST.COM
I20260927 15:12:56.236172 11447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:56.219861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45810 (local address 127.10.33.66:40893)
0927 15:12:56.220332 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:56.220337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:56.220357 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:12:56.220409 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:56.220413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:56.220486 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:56.220585 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:56.221960 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:56.222909 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:56.224289 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:56.224761 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:56.224896 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:56.229120 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:56.229149 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:56.229166 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:56.229208 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:56.232588 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:56.233448 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:56.233457 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:56.233459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:56.233612 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:56.234189 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:56.234194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:56.234197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:56.234690 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:56.234937 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:56.235357 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 15:12:56.235818 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":272,"threads_started":1}
I20260927 15:12:56.236860 11266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:12:56.328783 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 11143
I20260927 15:12:56.342087 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 11276
Sep 27 15:12:56 dist-test-slave-f5h6 krb5kdc[10903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.67@KRBTEST.COM
I20260927 15:12:56.356766 11421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:56.342820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35132 (local address 127.10.33.67:45909)
0927 15:12:56.342977 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:56.342981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:56.342999 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:12:56.343138 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:56.343142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:56.343200 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:56.343279 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:56.344484 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:56.345256 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:56.346692 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:56.347124 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:56.347225 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:56.351384 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:56.351402 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:56.351418 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:56.351464 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:56.354171 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:56.354870 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:56.354879 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:56.354888 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:56.355043 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:56.355405 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:56.355409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:56.355411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:56.355776 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:56.355994 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:56.356315 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 15:12:56.356496 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:12:56.357448 11399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:12:56.443061 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 11276
I20260927 15:12:56.455843 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 10916
I20260927 15:12:56.457031 10978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:12:56.562139 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 10916
2026-09-27T15:12:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4191 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:12:56 dist-test-slave-f5h6 krb5kdc[11453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:12:56 dist-test-slave-f5h6 krb5kdc[11453](info): set up 2 sockets
Sep 27 15:12:56 dist-test-slave-f5h6 krb5kdc[11453](info): commencing operation
krb5kdc: starting...
W20260927 15:12:57.565958 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:12:57 dist-test-slave-f5h6 krb5kdc[11453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:12:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:12:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:12:57.754478 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:44749
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:44749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:58.082643 11466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:58.083099 11466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:58.083249 11466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:58.091019 11466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:12:58.091156 11466 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 15:12:58.091199 11466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:58.091271 11466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:58.091351 11466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:12:58.091419 11466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:12:58.101948 11466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:44749
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:44749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:58.103262 11466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:58.104725 11466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:58.116742 11471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:58.116748 11474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:58.116941 11472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:58.118224 11466 server_base.cc:1034] running on GCE node
I20260927 15:12:58.119460 11466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:58.121034 11466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:58.122287 11466 hybrid_clock.cc:648] HybridClock initialized: now 1790521978122169 us; error 86 us; skew 500 ppm
I20260927 15:12:58.122727 11466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:58 dist-test-slave-f5h6 krb5kdc[11453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521978, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:58.126062 11466 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:12:58.127614 11466 webserver.cc:460] Webserver started at http://127.10.33.126:41359/ using document root <none> and password file <none>
I20260927 15:12:58.128334 11466 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:58.128437 11466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:58.128746 11466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:58.131781 11466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "1a9b3feda4204943ad83dc6938a8c8d2"
format_stamp: "Formatted at 2026-09-27 15:12:58 on dist-test-slave-f5h6"
server_key: "2d9383f9442b2f5fc43599ee00881612"
server_key_iv: "a3316744a67ba7abf1e82e86836a5241"
server_key_version: "encryptionkey@0"
I20260927 15:12:58.132540 11466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "1a9b3feda4204943ad83dc6938a8c8d2"
format_stamp: "Formatted at 2026-09-27 15:12:58 on dist-test-slave-f5h6"
server_key: "2d9383f9442b2f5fc43599ee00881612"
server_key_iv: "a3316744a67ba7abf1e82e86836a5241"
server_key_version: "encryptionkey@0"
I20260927 15:12:58.137848 11466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 15:12:58.142163 11481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:58.144512 11466 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:12:58.144824 11466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "1a9b3feda4204943ad83dc6938a8c8d2"
format_stamp: "Formatted at 2026-09-27 15:12:58 on dist-test-slave-f5h6"
server_key: "2d9383f9442b2f5fc43599ee00881612"
server_key_iv: "a3316744a67ba7abf1e82e86836a5241"
server_key_version: "encryptionkey@0"
I20260927 15:12:58.145226 11466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:58.166530 11466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:58.170933 11466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:58.171195 11466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:58.190701 11466 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:44749
I20260927 15:12:58.190729 11542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:44749 every 8 connection(s)
I20260927 15:12:58.192049 11466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:12:58.192728 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 11466
I20260927 15:12:58.192945 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:12:58.193653 10373 external_mini_cluster.cc:1428] Setting key 07b9a9d36e010575ee1fb3c42aa23c38
I20260927 15:12:58.196465 11543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 15:12:58 dist-test-slave-f5h6 krb5kdc[11453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:12:58.209494 11543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2: Bootstrap starting.
I20260927 15:12:58.213264 11543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:58.214622 11543 log.cc:824] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:58.217458 11543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2: No bootstrap required, opened a new log
I20260927 15:12:58.218031 11546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:58.195634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54722 (local address 127.10.33.126:44749)
0927 15:12:58.196385 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:58.196401 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:58.196464 (+ 63us) server_negotiation.cc:407] Connection header received
0927 15:12:58.198698 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:58.198727 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:58.199216 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:58.199721 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:58.201017 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:58.203112 (+ 2095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:58.204419 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:58.205436 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:58.205622 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:58.210318 (+ 4696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:58.210328 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:58.210345 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:58.210391 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:58.213565 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:58.214547 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:58.214554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:58.214562 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:58.214810 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:58.215213 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:58.215218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:58.215222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:58.215842 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:58.216127 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:58.216561 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 15:12:58.216967 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":303,"threads_started":1}
I20260927 15:12:58.225754 11543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9b3feda4204943ad83dc6938a8c8d2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44749 } }
I20260927 15:12:58.226188 11543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:12:58.226300 11543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9b3feda4204943ad83dc6938a8c8d2, State: Initialized, Role: FOLLOWER
I20260927 15:12:58.226895 11543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9b3feda4204943ad83dc6938a8c8d2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44749 } }
I20260927 15:12:58.227182 11543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:12:58.227316 11543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:12:58.227507 11543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:12:58.229753 11543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9b3feda4204943ad83dc6938a8c8d2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44749 } }
I20260927 15:12:58.230335 11543 leader_election.cc:302] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a9b3feda4204943ad83dc6938a8c8d2; no voters:
I20260927 15:12:58.230952 11543 leader_election.cc:288] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:12:58.231083 11548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:12:58.231734 11548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9b3feda4204943ad83dc6938a8c8d2, State: Running, Role: LEADER
I20260927 15:12:58.232264 11548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9b3feda4204943ad83dc6938a8c8d2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44749 } }
I20260927 15:12:58.232955 11543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:12:58.234915 11549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a9b3feda4204943ad83dc6938a8c8d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9b3feda4204943ad83dc6938a8c8d2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44749 } } }
I20260927 15:12:58.235132 11550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a9b3feda4204943ad83dc6938a8c8d2. Latest consensus state: current_term: 1 leader_uuid: "1a9b3feda4204943ad83dc6938a8c8d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9b3feda4204943ad83dc6938a8c8d2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44749 } } }
I20260927 15:12:58.235437 11550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2 [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:58.235471 11549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2 [sys.catalog]: This master's current role is: LEADER
I20260927 15:12:58.238437 11553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:12:58.244329 11553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:12:58.254180 11553 catalog_manager.cc:1347] Generated new cluster ID: d4b600b39a864090b228b9fee51d4a4e
I20260927 15:12:58.254310 11553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:12:58.272481 11553 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:12:58.274490 11553 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:12:58.288384 11553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2: Generated new TSK 0
I20260927 15:12:58.289537 11553 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.33.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
I20260927 15:12:58.339181 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:44749
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:58.661435 11569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:58.661759 11569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:58.661975 11569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:58.669350 11569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:58.669502 11569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:58.669622 11569 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:12:58.669732 11569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:12:58.680083 11569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.33.126:44749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:58.681387 11569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:58.682693 11569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:58.696278 11574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:58.696198 11577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:58.696244 11575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:58.697154 11569 server_base.cc:1034] running on GCE node
I20260927 15:12:58.698062 11569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:58.699064 11569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:58.700291 11569 hybrid_clock.cc:648] HybridClock initialized: now 1790521978700203 us; error 62 us; skew 500 ppm
I20260927 15:12:58.700637 11569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:58 dist-test-slave-f5h6 krb5kdc[11453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521978, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:58.703977 11569 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:12:58.705646 11569 webserver.cc:460] Webserver started at http://127.10.33.65:34477/ using document root <none> and password file <none>
I20260927 15:12:58.706421 11569 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:58.706529 11569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:58.706833 11569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:58.709784 11569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "e9b74eb460c448d4865dd19e74ffc346"
format_stamp: "Formatted at 2026-09-27 15:12:58 on dist-test-slave-f5h6"
server_key: "ff3f3fa4c7cb9898873d2a286417793f"
server_key_iv: "d9b7ed257638949274e28a203ccfbd40"
server_key_version: "encryptionkey@0"
I20260927 15:12:58.710687 11569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "e9b74eb460c448d4865dd19e74ffc346"
format_stamp: "Formatted at 2026-09-27 15:12:58 on dist-test-slave-f5h6"
server_key: "ff3f3fa4c7cb9898873d2a286417793f"
server_key_iv: "d9b7ed257638949274e28a203ccfbd40"
server_key_version: "encryptionkey@0"
I20260927 15:12:58.716264 11569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:12:58.720099 11584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:58.721549 11569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:12:58.721699 11569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "e9b74eb460c448d4865dd19e74ffc346"
format_stamp: "Formatted at 2026-09-27 15:12:58 on dist-test-slave-f5h6"
server_key: "ff3f3fa4c7cb9898873d2a286417793f"
server_key_iv: "d9b7ed257638949274e28a203ccfbd40"
server_key_version: "encryptionkey@0"
I20260927 15:12:58.721920 11569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:58.743306 11569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:58.747546 11569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:58.747769 11569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:58.750412 11569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:12:58.750496 11569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:58.750672 11569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:12:58.750727 11569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:12:58 dist-test-slave-f5h6 krb5kdc[11453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521978, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:12:58.775338 11594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:58.751752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:44749 (local address 127.10.33.65:36045)
0927 15:12:58.752574 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:58.752621 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:58.754616 (+ 1995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:58.755489 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:58.755502 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:58.756245 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:58.757163 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:58.757185 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:58.758654 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:58.758659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:58.760579 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:58.760597 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:58.761713 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:58.761718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:58.762069 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:58.762834 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:58.762863 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:58.766248 (+ 3385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:58.770192 (+ 3944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:58.770208 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:58.770231 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:58.771033 (+ 802us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:58.771462 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:58.771468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:58.771472 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:58.771771 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:58.772462 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:58.772474 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:58.772886 (+ 412us) client_negotiation.cc:769] Sending connection context
0927 15:12:58.773173 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 15:12:58.773742 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":338,"threads_started":1}
I20260927 15:12:58.776666 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:58.751061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:36045 (local address 127.10.33.126:44749)
0927 15:12:58.751530 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:58.751536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:58.752748 (+ 1212us) server_negotiation.cc:407] Connection header received
0927 15:12:58.754952 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:58.754958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:58.755054 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:58.755184 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:58.757347 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:58.758490 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:58.760721 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:58.761273 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:58.761400 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:58.766592 (+ 5192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:58.766600 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:58.766606 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:58.766656 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:58.770007 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:58.771175 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:58.771181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:58.771185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:58.771340 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:58.771906 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:58.771911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:58.771914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:58.772236 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:58.772379 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:58.776156 (+ 3777us) server_negotiation.cc:299] Negotiation successful
0927 15:12:58.776334 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":251,"threads_started":1}
I20260927 15:12:58.792157 11569 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:41025
I20260927 15:12:58.792182 11704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:41025 every 8 connection(s)
I20260927 15:12:58.793498 11569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:12:58.795558 11705 heartbeater.cc:347] Connected to a master server at 127.10.33.126:44749
I20260927 15:12:58.795770 11705 heartbeater.cc:464] Registering TS with master...
I20260927 15:12:58.796427 11705 heartbeater.cc:511] Master 127.10.33.126:44749 requested a full tablet report, sending...
I20260927 15:12:58.796718 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:58.781849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:58387 (local address 127.10.33.126:44749)
0927 15:12:58.782073 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:58.782078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:58.782738 (+ 660us) server_negotiation.cc:407] Connection header received
0927 15:12:58.782900 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:58.782905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:58.783019 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:58.783139 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:58.784992 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:58.786057 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:58.788159 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:58.788731 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:58.788848 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:58.791225 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:58.791232 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:58.791236 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:58.791275 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:58.793963 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:58.794834 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:58.794839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:58.794852 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:58.795012 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:58.795383 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:58.795387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:58.795390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:58.795685 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:58.795818 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:58.796193 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 15:12:58.796390 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:12:58.796839 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:58.781784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:44749 (local address 127.10.33.65:58387)
0927 15:12:58.782626 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:58.782648 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:58.782776 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:58.783272 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:58.783278 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:58.784312 (+ 1034us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:58.784743 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:58.784762 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:58.786200 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:58.786206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:58.788013 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:58.788031 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:58.788952 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:58.788962 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:58.789115 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:58.790035 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:58.790052 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:58.790985 (+ 933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:58.794088 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:58.794093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:58.794097 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:58.794680 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:58.795127 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:58.795131 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:58.795133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:58.795275 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:58.795787 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:58.795792 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:58.795962 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 15:12:58.796079 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 15:12:58.796419 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":569,"threads_started":1}
I20260927 15:12:58.797514 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 11569
I20260927 15:12:58.797719 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:12:58.798524 10373 external_mini_cluster.cc:1428] Setting key d515158eede1b2b2ad1700024e3d5315
I20260927 15:12:58.798528 11497 ts_manager.cc:194] Registered new tserver with Master: e9b74eb460c448d4865dd19e74ffc346 (127.10.33.65:41025)
I20260927 15:12:58.800024 11497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:36045
WARNING: no policy specified for kudu/127.10.33.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
I20260927 15:12:58.838496 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:44749
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:59.158586 11709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:59.158905 11709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:59.159044 11709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:59.167047 11709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:59.167196 11709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:59.167311 11709 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:12:59.167438 11709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:12:59.177721 11709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.33.126:44749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:59.179123 11709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:59.180517 11709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:59.194571 11714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:59.194608 11715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:59.194634 11717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:59.196283 11709 server_base.cc:1034] running on GCE node
I20260927 15:12:59.196947 11709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:59.197971 11709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:59.199187 11709 hybrid_clock.cc:648] HybridClock initialized: now 1790521979199107 us; error 61 us; skew 500 ppm
I20260927 15:12:59.199513 11709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:59 dist-test-slave-f5h6 krb5kdc[11453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521979, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:59.207618 11709 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:12:59.209227 11709 webserver.cc:460] Webserver started at http://127.10.33.66:34733/ using document root <none> and password file <none>
I20260927 15:12:59.210016 11709 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:59.210129 11709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:59.210431 11709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:59.213374 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "793d49b1ae01412aaa22f1172a2854a8"
format_stamp: "Formatted at 2026-09-27 15:12:59 on dist-test-slave-f5h6"
server_key: "c22cbd418d5dbd2a4f9dfc864ebffbc8"
server_key_iv: "4528a4475b9c9918edb1f5393f7a7454"
server_key_version: "encryptionkey@0"
I20260927 15:12:59.214228 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "793d49b1ae01412aaa22f1172a2854a8"
format_stamp: "Formatted at 2026-09-27 15:12:59 on dist-test-slave-f5h6"
server_key: "c22cbd418d5dbd2a4f9dfc864ebffbc8"
server_key_iv: "4528a4475b9c9918edb1f5393f7a7454"
server_key_version: "encryptionkey@0"
I20260927 15:12:59.219473 11709 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 15:12:59.223026 11724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:59.224560 11709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:12:59.224721 11709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "793d49b1ae01412aaa22f1172a2854a8"
format_stamp: "Formatted at 2026-09-27 15:12:59 on dist-test-slave-f5h6"
server_key: "c22cbd418d5dbd2a4f9dfc864ebffbc8"
server_key_iv: "4528a4475b9c9918edb1f5393f7a7454"
server_key_version: "encryptionkey@0"
I20260927 15:12:59.224933 11709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:59.237581 11709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:59.241719 11709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:59.241993 11709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:59.244686 11709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:12:59.244769 11709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:59.244920 11709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:12:59.244977 11709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:12:59 dist-test-slave-f5h6 krb5kdc[11453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521979, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:12:59.269392 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:59.245307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:34821 (local address 127.10.33.126:44749)
0927 15:12:59.245525 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:59.245533 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:59.247003 (+ 1470us) server_negotiation.cc:407] Connection header received
0927 15:12:59.249806 (+ 2803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:59.249813 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:59.249956 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:59.250041 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:59.252954 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.253979 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.256433 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.257077 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.257235 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:59.262830 (+ 5595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:59.262839 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:59.262844 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:59.262894 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:59.266185 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:59.266981 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:59.266987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:59.266992 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:59.267149 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:59.267531 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:59.267536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:59.267539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:59.267915 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:59.268059 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:59.268754 (+ 695us) server_negotiation.cc:299] Negotiation successful
0927 15:12:59.268997 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:12:59.270325 11733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:59.246031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:44749 (local address 127.10.33.66:34821)
0927 15:12:59.246812 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:59.246867 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:59.249436 (+ 2569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:59.250653 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:59.250667 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:59.251548 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:59.252750 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:59.252779 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.254196 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.254201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:59.256273 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:59.256299 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.257381 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.257390 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:59.257705 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:59.258688 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:59.258727 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:59.262473 (+ 3746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:59.266366 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:59.266377 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:59.266398 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:59.266847 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:59.267249 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:59.267252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:59.267254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:59.267444 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:59.268039 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:59.268045 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:59.268408 (+ 363us) client_negotiation.cc:769] Sending connection context
0927 15:12:59.268646 (+ 238us) client_negotiation.cc:240] Negotiation successful
0927 15:12:59.269052 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":304,"threads_started":1}
I20260927 15:12:59.291463 11709 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:36703
I20260927 15:12:59.291503 11843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:36703 every 8 connection(s)
I20260927 15:12:59.291877 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:59.277390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:42485 (local address 127.10.33.126:44749)
0927 15:12:59.277591 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:59.277598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:59.277731 (+ 133us) server_negotiation.cc:407] Connection header received
0927 15:12:59.277985 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:59.277991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:59.278072 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:59.278153 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:59.279526 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.280511 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.282564 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.282970 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.283082 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:59.285813 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:59.285819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:59.285823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:59.285887 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:59.288917 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:59.289747 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:59.289753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:59.289757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:59.289932 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:59.290283 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:59.290289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:59.290293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:59.290639 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:59.290779 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:59.291350 (+ 571us) server_negotiation.cc:299] Negotiation successful
0927 15:12:59.291553 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:12:59.292062 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:59.276973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:44749 (local address 127.10.33.66:42485)
0927 15:12:59.277641 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:59.277657 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:59.277827 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:59.278202 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:59.278210 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:59.278879 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:59.279329 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:59.279343 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.280719 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.280725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:59.282418 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:59.282434 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.283164 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.283170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:59.283337 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:59.284216 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:59.284238 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:59.285533 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:59.289050 (+ 3517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:59.289056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:59.289064 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:59.289628 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:59.290036 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:59.290041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:59.290045 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:59.290183 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:59.290850 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:59.290854 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:59.291065 (+ 211us) client_negotiation.cc:769] Sending connection context
0927 15:12:59.291234 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 15:12:59.291626 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":460,"threads_started":1}
I20260927 15:12:59.292989 11709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:12:59.295070 11844 heartbeater.cc:347] Connected to a master server at 127.10.33.126:44749
I20260927 15:12:59.295303 11844 heartbeater.cc:464] Registering TS with master...
I20260927 15:12:59.295979 11844 heartbeater.cc:511] Master 127.10.33.126:44749 requested a full tablet report, sending...
I20260927 15:12:59.296696 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 11709
I20260927 15:12:59.296867 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:12:59.297266 10373 external_mini_cluster.cc:1428] Setting key e806976ba777970065b7d6ac6495d1e2
I20260927 15:12:59.297247 11497 ts_manager.cc:194] Registered new tserver with Master: 793d49b1ae01412aaa22f1172a2854a8 (127.10.33.66:36703)
I20260927 15:12:59.298107 11497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:34821
WARNING: no policy specified for kudu/127.10.33.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
I20260927 15:12:59.336974 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:44749
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:12:59.671931 11848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:12:59.672245 11848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:12:59.672362 11848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:12:59.680071 11848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:12:59.680199 11848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:12:59.680308 11848 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:12:59.680405 11848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:12:59.690713 11848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.33.126:44749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:12:59.692034 11848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:12:59.693358 11848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:12:59.706179 11854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:59.706162 11853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:12:59.706382 11856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:12:59.707160 11848 server_base.cc:1034] running on GCE node
I20260927 15:12:59.707763 11848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:12:59.708668 11848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:12:59.709955 11848 hybrid_clock.cc:648] HybridClock initialized: now 1790521979709861 us; error 49 us; skew 500 ppm
I20260927 15:12:59.710305 11848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:12:59 dist-test-slave-f5h6 krb5kdc[11453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521979, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:12:59.713665 11848 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:12:59.715217 11848 webserver.cc:460] Webserver started at http://127.10.33.67:34129/ using document root <none> and password file <none>
I20260927 15:12:59.715911 11848 fs_manager.cc:359] Metadata directory not provided
I20260927 15:12:59.715996 11848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:12:59.716293 11848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:12:59.719233 11848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "2b0da842519a49f881bf9c43ea34941f"
format_stamp: "Formatted at 2026-09-27 15:12:59 on dist-test-slave-f5h6"
server_key: "346aaadf4a5e2258daae69f84f43d8a4"
server_key_iv: "8336857c377cd4aa77ada29f3952bfff"
server_key_version: "encryptionkey@0"
I20260927 15:12:59.720022 11848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "2b0da842519a49f881bf9c43ea34941f"
format_stamp: "Formatted at 2026-09-27 15:12:59 on dist-test-slave-f5h6"
server_key: "346aaadf4a5e2258daae69f84f43d8a4"
server_key_iv: "8336857c377cd4aa77ada29f3952bfff"
server_key_version: "encryptionkey@0"
I20260927 15:12:59.725416 11848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 15:12:59.729214 11863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:59.730794 11848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 15:12:59.730954 11848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "2b0da842519a49f881bf9c43ea34941f"
format_stamp: "Formatted at 2026-09-27 15:12:59 on dist-test-slave-f5h6"
server_key: "346aaadf4a5e2258daae69f84f43d8a4"
server_key_iv: "8336857c377cd4aa77ada29f3952bfff"
server_key_version: "encryptionkey@0"
I20260927 15:12:59.731158 11848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:12:59.745041 11848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:12:59.748955 11848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:12:59.749200 11848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:12:59.751843 11848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:12:59.751928 11848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:12:59.752070 11848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:12:59.752141 11848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:12:59 dist-test-slave-f5h6 krb5kdc[11453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521979, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:12:59.777457 11872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:59.753418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:44749 (local address 127.10.33.67:59779)
0927 15:12:59.754238 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:59.754281 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:59.756128 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:59.756943 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:59.756952 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:59.757694 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:59.758630 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:59.758658 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.760020 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.760027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:59.762652 (+ 2625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:59.762690 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.763718 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.763727 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:59.764115 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:59.765072 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:59.765111 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:59.768316 (+ 3205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:59.772348 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:59.772359 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:59.772384 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:59.773047 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:59.773449 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:59.773455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:59.773459 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:59.773684 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:59.774338 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:59.774347 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:59.774920 (+ 573us) client_negotiation.cc:769] Sending connection context
0927 15:12:59.775282 (+ 362us) client_negotiation.cc:240] Negotiation successful
0927 15:12:59.775848 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":370,"threads_started":1}
I20260927 15:12:59.778491 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:59.752709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:59779 (local address 127.10.33.126:44749)
0927 15:12:59.752874 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:59.752880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:59.754392 (+ 1512us) server_negotiation.cc:407] Connection header received
0927 15:12:59.756511 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:59.756518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:59.756621 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:59.756742 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:59.758832 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.759788 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.762897 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.763424 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.763570 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:59.768649 (+ 5079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:59.768657 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:59.768662 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:59.768741 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:59.772176 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:59.773182 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:59.773188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:59.773191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:59.773342 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:59.773805 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:59.773810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:59.773814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:59.774217 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:59.774359 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:59.777958 (+ 3599us) server_negotiation.cc:299] Negotiation successful
0927 15:12:59.778155 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:12:59.797684 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:59.784127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:44749 (local address 127.10.33.67:35911)
0927 15:12:59.784690 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:59.784709 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:59.784852 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:59.785187 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:59.785192 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:59.785672 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:12:59.786045 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:59.786056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.787001 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.787007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:59.788013 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:59.788025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.788717 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.788724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:59.788882 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:59.789496 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:12:59.789523 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:12:59.790696 (+ 1173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:12:59.794418 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:59.794429 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:59.794433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:59.795021 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:59.795611 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:12:59.795616 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:12:59.795620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:12:59.795770 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:12:59.796412 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:12:59.796419 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:12:59.796653 (+ 234us) client_negotiation.cc:769] Sending connection context
0927 15:12:59.796789 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 15:12:59.797163 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":315,"threads_started":1}
I20260927 15:12:59.798779 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:59.784033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:35911 (local address 127.10.33.126:44749)
0927 15:12:59.784183 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:59.784187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:59.784794 (+ 607us) server_negotiation.cc:407] Connection header received
0927 15:12:59.784989 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:59.784995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:59.785067 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:59.785145 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:59.786181 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.786876 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.788137 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.788549 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.788652 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:59.790878 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:59.790885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:59.790888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:59.790921 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:59.794232 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:59.795315 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:59.795321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:59.795324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:59.795481 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:59.795889 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:59.795894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:59.795897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:59.796263 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:59.798025 (+ 1762us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:59.798166 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 15:12:59.798416 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:12:59.799947 11848 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:43983
I20260927 15:12:59.800005 11982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:43983 every 8 connection(s)
I20260927 15:12:59.801224 11848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:12:59.803184 11705 heartbeater.cc:503] Master 127.10.33.126:44749 was elected leader, sending a full tablet report...
I20260927 15:12:59.803792 11983 heartbeater.cc:347] Connected to a master server at 127.10.33.126:44749
I20260927 15:12:59.804045 11983 heartbeater.cc:464] Registering TS with master...
I20260927 15:12:59.804175 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 11848
I20260927 15:12:59.804421 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:12:59.804826 11983 heartbeater.cc:511] Master 127.10.33.126:44749 requested a full tablet report, sending...
I20260927 15:12:59.804905 10373 external_mini_cluster.cc:1428] Setting key 1e4080f560740872f08443d26569f28e
I20260927 15:12:59.806213 11498 ts_manager.cc:194] Registered new tserver with Master: 2b0da842519a49f881bf9c43ea34941f (127.10.33.67:43983)
I20260927 15:12:59.807133 11498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:59779
I20260927 15:12:59.807726 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:12:59 dist-test-slave-f5h6 krb5kdc[11453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521979, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:12:59 dist-test-slave-f5h6 krb5kdc[11453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521979, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:12:59.852239 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:59.838191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54724 (local address 127.10.33.126:44749)
0927 15:12:59.838418 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:59.838424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:59.838759 (+ 335us) server_negotiation.cc:407] Connection header received
0927 15:12:59.838956 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:59.838961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:59.839030 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:59.839121 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:12:59.840529 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.841261 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.842692 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.843131 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.843230 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:59.847393 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:12:59.847399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:12:59.847402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:12:59.847467 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:12:59.849969 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:59.850584 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:59.850588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:59.850590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:59.850707 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:12:59.851059 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:12:59.851062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:12:59.851064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:12:59.851326 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:12:59.851512 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:59.851736 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 15:12:59.851922 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 15:12:59.856293 11498 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:12:59.863281 11498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:12:59.888032 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:59.876126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:41025 (local address 127.0.0.1:48856)
0927 15:12:59.877055 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:59.877092 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:59.877229 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:59.878548 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:59.878558 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:59.878594 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:12:59.879233 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:59.879266 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.881277 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.881285 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:59.884090 (+ 2805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:59.884107 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.886797 (+ 2690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.886806 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:59.887083 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:59.887148 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 15:12:59.887326 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 15:12:59.887574 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":329,"threads_started":1}
I20260927 15:12:59.888803 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:59.877538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:36703 (local address 127.0.0.1:33486)
0927 15:12:59.879261 (+ 1723us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:59.879280 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:59.879397 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:59.880420 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:59.880426 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:59.880454 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:12:59.880887 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:59.880899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.883032 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.883039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:59.885478 (+ 2439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:59.885498 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.887920 (+ 2422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.887927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:59.888136 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:59.888171 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 15:12:59.888259 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 15:12:59.888409 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1606,"thread_start_us":1401,"threads_started":1}
I20260927 15:12:59.889546 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:59.876363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48856 (local address 127.10.33.65:41025)
0927 15:12:59.877752 (+ 1389us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:59.877762 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:59.877788 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:12:59.877947 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:59.877957 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:59.878268 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:59.878524 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:12:59.879508 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.881050 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.884340 (+ 3290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.886181 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.886370 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:59.886639 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:59.888936 (+ 2297us) server_negotiation.cc:299] Negotiation successful
0927 15:12:59.889131 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1243,"thread_start_us":1108,"threads_started":1}
I20260927 15:12:59.889783 11999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:59.878038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33486 (local address 127.10.33.66:36703)
0927 15:12:59.878514 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:59.878522 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:59.879518 (+ 996us) server_negotiation.cc:407] Connection header received
0927 15:12:59.879620 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:59.879628 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:59.879921 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:59.880129 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:12:59.881089 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.882824 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.885703 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.887393 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.887538 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:59.887747 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:59.889274 (+ 1527us) server_negotiation.cc:299] Negotiation successful
0927 15:12:59.889478 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":214,"threads_started":1}
I20260927 15:12:59.893988 11774 tablet_service.cc:1467] Processing CreateTablet for tablet 21ac147ec412413c8bfe10ee1abfcd6e (DEFAULT_TABLE table=test-table [id=c48bf2b642f948ef83759b4cecdeb558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:12:59.893983 11635 tablet_service.cc:1467] Processing CreateTablet for tablet 21ac147ec412413c8bfe10ee1abfcd6e (DEFAULT_TABLE table=test-table [id=c48bf2b642f948ef83759b4cecdeb558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:12:59.895309 11635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21ac147ec412413c8bfe10ee1abfcd6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:59.895319 11774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21ac147ec412413c8bfe10ee1abfcd6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:59.905967 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:59.879539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:43983 (local address 127.0.0.1:54576)
0927 15:12:59.888600 (+ 9061us) negotiation.cc:107] Waiting for socket to connect
0927 15:12:59.888621 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:12:59.888733 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:12:59.891538 (+ 2805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:12:59.891544 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:12:59.891576 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:12:59.891756 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:59.891768 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.901003 (+ 9235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.901010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:59.902914 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:12:59.902935 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.905090 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.905097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:12:59.905292 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:59.905338 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 15:12:59.905463 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 15:12:59.905628 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8958,"thread_start_us":10778,"threads_started":1}
I20260927 15:12:59.906044 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:12:59.879848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54576 (local address 127.10.33.67:43983)
0927 15:12:59.890785 (+ 10937us) server_negotiation.cc:206] Beginning negotiation
0927 15:12:59.890795 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:12:59.890832 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:12:59.890949 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:12:59.890958 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:12:59.891240 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:12:59.891485 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:12:59.898213 (+ 6728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.899849 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.903135 (+ 3286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:12:59.904866 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:12:59.905034 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:12:59.905286 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 15:12:59.905510 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 15:12:59.905688 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10772,"thread_start_us":10607,"threads_started":1}
I20260927 15:12:59.909065 12003 tablet_bootstrap.cc:492] T 21ac147ec412413c8bfe10ee1abfcd6e P 793d49b1ae01412aaa22f1172a2854a8: Bootstrap starting.
I20260927 15:12:59.910457 11913 tablet_service.cc:1467] Processing CreateTablet for tablet 21ac147ec412413c8bfe10ee1abfcd6e (DEFAULT_TABLE table=test-table [id=c48bf2b642f948ef83759b4cecdeb558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:12:59.911904 11913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21ac147ec412413c8bfe10ee1abfcd6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:12:59.913211 12003 tablet_bootstrap.cc:654] T 21ac147ec412413c8bfe10ee1abfcd6e P 793d49b1ae01412aaa22f1172a2854a8: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:59.914505 12002 tablet_bootstrap.cc:492] T 21ac147ec412413c8bfe10ee1abfcd6e P e9b74eb460c448d4865dd19e74ffc346: Bootstrap starting.
I20260927 15:12:59.914943 12003 log.cc:824] T 21ac147ec412413c8bfe10ee1abfcd6e P 793d49b1ae01412aaa22f1172a2854a8: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:59.918165 12002 tablet_bootstrap.cc:654] T 21ac147ec412413c8bfe10ee1abfcd6e P e9b74eb460c448d4865dd19e74ffc346: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:59.919459 12003 tablet_bootstrap.cc:492] T 21ac147ec412413c8bfe10ee1abfcd6e P 793d49b1ae01412aaa22f1172a2854a8: No bootstrap required, opened a new log
I20260927 15:12:59.919449 12002 log.cc:824] T 21ac147ec412413c8bfe10ee1abfcd6e P e9b74eb460c448d4865dd19e74ffc346: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:59.919745 12003 ts_tablet_manager.cc:1397] T 21ac147ec412413c8bfe10ee1abfcd6e P 793d49b1ae01412aaa22f1172a2854a8: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260927 15:12:59.927161 12002 tablet_bootstrap.cc:492] T 21ac147ec412413c8bfe10ee1abfcd6e P e9b74eb460c448d4865dd19e74ffc346: No bootstrap required, opened a new log
I20260927 15:12:59.927575 12002 ts_tablet_manager.cc:1397] T 21ac147ec412413c8bfe10ee1abfcd6e P e9b74eb460c448d4865dd19e74ffc346: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.007s
I20260927 15:12:59.930666 12003 raft_consensus.cc:348] T 21ac147ec412413c8bfe10ee1abfcd6e P 793d49b1ae01412aaa22f1172a2854a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:12:59.931191 12003 raft_consensus.cc:374] T 21ac147ec412413c8bfe10ee1abfcd6e P 793d49b1ae01412aaa22f1172a2854a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:12:59.931348 12003 raft_consensus.cc:729] T 21ac147ec412413c8bfe10ee1abfcd6e P 793d49b1ae01412aaa22f1172a2854a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793d49b1ae01412aaa22f1172a2854a8, State: Initialized, Role: FOLLOWER
I20260927 15:12:59.932072 12003 consensus_queue.cc:260] T 21ac147ec412413c8bfe10ee1abfcd6e P 793d49b1ae01412aaa22f1172a2854a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:12:59.933233 11844 heartbeater.cc:503] Master 127.10.33.126:44749 was elected leader, sending a full tablet report...
I20260927 15:12:59.933976 12003 ts_tablet_manager.cc:1428] T 21ac147ec412413c8bfe10ee1abfcd6e P 793d49b1ae01412aaa22f1172a2854a8: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260927 15:12:59.932967 12006 tablet_bootstrap.cc:492] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f: Bootstrap starting.
I20260927 15:12:59.936153 12002 raft_consensus.cc:348] T 21ac147ec412413c8bfe10ee1abfcd6e P e9b74eb460c448d4865dd19e74ffc346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:12:59.936685 12002 raft_consensus.cc:374] T 21ac147ec412413c8bfe10ee1abfcd6e P e9b74eb460c448d4865dd19e74ffc346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:12:59.936792 12002 raft_consensus.cc:729] T 21ac147ec412413c8bfe10ee1abfcd6e P e9b74eb460c448d4865dd19e74ffc346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b74eb460c448d4865dd19e74ffc346, State: Initialized, Role: FOLLOWER
I20260927 15:12:59.937419 12002 consensus_queue.cc:260] T 21ac147ec412413c8bfe10ee1abfcd6e P e9b74eb460c448d4865dd19e74ffc346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:12:59.939133 12002 ts_tablet_manager.cc:1428] T 21ac147ec412413c8bfe10ee1abfcd6e P e9b74eb460c448d4865dd19e74ffc346: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260927 15:12:59.939162 12006 tablet_bootstrap.cc:654] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f: Neither blocks nor log segments found. Creating new log.
I20260927 15:12:59.940534 12006 log.cc:824] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:12:59.943688 12006 tablet_bootstrap.cc:492] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f: No bootstrap required, opened a new log
I20260927 15:12:59.943945 12006 ts_tablet_manager.cc:1397] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260927 15:12:59.950956 12006 raft_consensus.cc:348] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:12:59.951334 12006 raft_consensus.cc:374] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:12:59.951398 12006 raft_consensus.cc:729] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0da842519a49f881bf9c43ea34941f, State: Initialized, Role: FOLLOWER
I20260927 15:12:59.951933 12006 consensus_queue.cc:260] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:12:59.952944 11983 heartbeater.cc:503] Master 127.10.33.126:44749 was elected leader, sending a full tablet report...
I20260927 15:12:59.953416 12006 ts_tablet_manager.cc:1428] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
W20260927 15:13:00.045611 11845 tablet.cc:2369] T 21ac147ec412413c8bfe10ee1abfcd6e P 793d49b1ae01412aaa22f1172a2854a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:13:00.045944 11706 tablet.cc:2369] T 21ac147ec412413c8bfe10ee1abfcd6e P e9b74eb460c448d4865dd19e74ffc346: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:13:00.052961 11984 tablet.cc:2369] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:13:00.075588 12010 raft_consensus.cc:482] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:13:00.075946 12010 raft_consensus.cc:504] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:13:00.077582 12010 leader_election.cc:288] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9b74eb460c448d4865dd19e74ffc346 (127.10.33.65:41025), 793d49b1ae01412aaa22f1172a2854a8 (127.10.33.66:36703)
I20260927 15:13:00.084565 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:00.078007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:48723 (local address 127.10.33.65:41025)
0927 15:13:00.078193 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:00.078199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:00.078330 (+ 131us) server_negotiation.cc:407] Connection header received
0927 15:13:00.078657 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:00.078663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:00.078754 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:00.078859 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:00.079370 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.080393 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.082648 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.083745 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.083914 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:00.083992 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:00.084203 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 15:13:00.084329 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:13:00.084564 11872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:00.078033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:41025 (local address 127.10.33.67:48723)
0927 15:13:00.078246 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:00.078267 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:00.078489 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:00.078995 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:00.079000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:00.079030 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:00.079199 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:00.079211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.080522 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.080526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:00.082466 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:00.082481 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.083891 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.083895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:00.084058 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:00.084083 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:13:00.084156 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 15:13:00.084252 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":29}
I20260927 15:13:00.085124 11999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:00.078623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:60841 (local address 127.10.33.66:36703)
0927 15:13:00.078864 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:00.078870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:00.079237 (+ 367us) server_negotiation.cc:407] Connection header received
0927 15:13:00.079392 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:00.079397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:00.079468 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:00.079553 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:00.080013 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.080917 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.083079 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.084222 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.084373 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:00.084439 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:00.084769 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 15:13:00.084899 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 15:13:00.085649 12011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:00.078428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:36703 (local address 127.10.33.67:60841)
0927 15:13:00.079131 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:00.079148 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:00.079271 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:00.079573 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:00.079576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:00.079593 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:00.079887 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:00.079896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.081034 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.081039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:00.082927 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:00.082937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.084432 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.084439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:00.084590 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:00.084621 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 15:13:00.085267 (+ 646us) client_negotiation.cc:240] Negotiation successful
0927 15:13:00.085384 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":1,"thread_start_us":388,"threads_started":1}
I20260927 15:13:00.085642 11655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21ac147ec412413c8bfe10ee1abfcd6e" candidate_uuid: "2b0da842519a49f881bf9c43ea34941f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9b74eb460c448d4865dd19e74ffc346" is_pre_election: true
I20260927 15:13:00.086251 11655 raft_consensus.cc:2454] T 21ac147ec412413c8bfe10ee1abfcd6e P e9b74eb460c448d4865dd19e74ffc346 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b0da842519a49f881bf9c43ea34941f in term 0.
I20260927 15:13:00.086639 11795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21ac147ec412413c8bfe10ee1abfcd6e" candidate_uuid: "2b0da842519a49f881bf9c43ea34941f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793d49b1ae01412aaa22f1172a2854a8" is_pre_election: true
I20260927 15:13:00.086853 11865 leader_election.cc:302] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0da842519a49f881bf9c43ea34941f, e9b74eb460c448d4865dd19e74ffc346; no voters:
I20260927 15:13:00.087131 11795 raft_consensus.cc:2454] T 21ac147ec412413c8bfe10ee1abfcd6e P 793d49b1ae01412aaa22f1172a2854a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b0da842519a49f881bf9c43ea34941f in term 0.
I20260927 15:13:00.087237 12010 raft_consensus.cc:2781] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:13:00.087378 12010 raft_consensus.cc:482] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:13:00.087487 12010 raft_consensus.cc:3037] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:00.089607 12010 raft_consensus.cc:504] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:13:00.090195 12010 leader_election.cc:288] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f [CANDIDATE]: Term 1 election: Requested vote from peers e9b74eb460c448d4865dd19e74ffc346 (127.10.33.65:41025), 793d49b1ae01412aaa22f1172a2854a8 (127.10.33.66:36703)
I20260927 15:13:00.090730 11655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21ac147ec412413c8bfe10ee1abfcd6e" candidate_uuid: "2b0da842519a49f881bf9c43ea34941f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9b74eb460c448d4865dd19e74ffc346"
I20260927 15:13:00.090909 11655 raft_consensus.cc:3037] T 21ac147ec412413c8bfe10ee1abfcd6e P e9b74eb460c448d4865dd19e74ffc346 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:00.090893 11795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21ac147ec412413c8bfe10ee1abfcd6e" candidate_uuid: "2b0da842519a49f881bf9c43ea34941f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793d49b1ae01412aaa22f1172a2854a8"
I20260927 15:13:00.091102 11795 raft_consensus.cc:3037] T 21ac147ec412413c8bfe10ee1abfcd6e P 793d49b1ae01412aaa22f1172a2854a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:00.093091 11655 raft_consensus.cc:2454] T 21ac147ec412413c8bfe10ee1abfcd6e P e9b74eb460c448d4865dd19e74ffc346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0da842519a49f881bf9c43ea34941f in term 1.
I20260927 15:13:00.093510 11865 leader_election.cc:302] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0da842519a49f881bf9c43ea34941f, e9b74eb460c448d4865dd19e74ffc346; no voters:
I20260927 15:13:00.093595 11795 raft_consensus.cc:2454] T 21ac147ec412413c8bfe10ee1abfcd6e P 793d49b1ae01412aaa22f1172a2854a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0da842519a49f881bf9c43ea34941f in term 1.
I20260927 15:13:00.093813 12010 raft_consensus.cc:2781] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:00.094451 12010 raft_consensus.cc:686] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f [term 1 LEADER]: Becoming Leader. State: Replica: 2b0da842519a49f881bf9c43ea34941f, State: Running, Role: LEADER
I20260927 15:13:00.095027 12010 consensus_queue.cc:237] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:13:00.099802 11497 catalog_manager.cc:5495] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b0da842519a49f881bf9c43ea34941f (127.10.33.67). New cstate: current_term: 1 leader_uuid: "2b0da842519a49f881bf9c43ea34941f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } health_report { overall_health: HEALTHY } } }
I20260927 15:13:00.134387 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:00.128482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54588 (local address 127.10.33.67:43983)
0927 15:13:00.128715 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:00.128721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:00.128895 (+ 174us) server_negotiation.cc:407] Connection header received
0927 15:13:00.129229 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:00.129234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:00.129321 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:00.129464 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:13:00.129812 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.130569 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.131945 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.132443 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.132579 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:00.133144 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:13:00.133746 (+ 602us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:13:00.133936 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:00.134038 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 15:13:00.134120 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 15:13:00.142897 11795 raft_consensus.cc:1261] T 21ac147ec412413c8bfe10ee1abfcd6e P 793d49b1ae01412aaa22f1172a2854a8 [term 1 FOLLOWER]: Refusing update from remote peer 2b0da842519a49f881bf9c43ea34941f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:00.142897 11655 raft_consensus.cc:1261] T 21ac147ec412413c8bfe10ee1abfcd6e P e9b74eb460c448d4865dd19e74ffc346 [term 1 FOLLOWER]: Refusing update from remote peer 2b0da842519a49f881bf9c43ea34941f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:00.143983 12012 consensus_queue.cc:1035] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:00.144409 12010 consensus_queue.cc:1035] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f [LEADER]: Connected to new peer: Peer: permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:00.179680 11498 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:00.179951 11498 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:00.183677 11498 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2: Sending DeleteTablet for 3 replicas of tablet 21ac147ec412413c8bfe10ee1abfcd6e
I20260927 15:13:00.185106 11635 tablet_service.cc:1514] Processing DeleteTablet for tablet 21ac147ec412413c8bfe10ee1abfcd6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:00 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:48856
I20260927 15:13:00.185506 11913 tablet_service.cc:1514] Processing DeleteTablet for tablet 21ac147ec412413c8bfe10ee1abfcd6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:00 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:54576
I20260927 15:13:00.186050 11774 tablet_service.cc:1514] Processing DeleteTablet for tablet 21ac147ec412413c8bfe10ee1abfcd6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:00 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:33486
I20260927 15:13:00.186590 11497 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:13:00.187374 11497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:13:00.189093 12028 tablet_replica.cc:330] T 21ac147ec412413c8bfe10ee1abfcd6e P 793d49b1ae01412aaa22f1172a2854a8: stopping tablet replica
I20260927 15:13:00.189606 12028 raft_consensus.cc:2229] T 21ac147ec412413c8bfe10ee1abfcd6e P 793d49b1ae01412aaa22f1172a2854a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:00.190143 12028 raft_consensus.cc:2258] T 21ac147ec412413c8bfe10ee1abfcd6e P 793d49b1ae01412aaa22f1172a2854a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:00.199677 11634 tablet_service.cc:1467] Processing CreateTablet for tablet f6f959d3dc29465289a18a5dc65be02a (DEFAULT_TABLE table=test-table [id=5e641ebbf7bd433e819b78f00bc7ce4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:00.200264 11634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f959d3dc29465289a18a5dc65be02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:00.200781 12027 tablet_replica.cc:330] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f: stopping tablet replica
I20260927 15:13:00.201284 12027 raft_consensus.cc:2229] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:13:00.201428 11912 tablet_service.cc:1467] Processing CreateTablet for tablet f6f959d3dc29465289a18a5dc65be02a (DEFAULT_TABLE table=test-table [id=5e641ebbf7bd433e819b78f00bc7ce4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:00.202016 12027 raft_consensus.cc:2258] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:00.202483 11774 tablet_service.cc:1467] Processing CreateTablet for tablet f6f959d3dc29465289a18a5dc65be02a (DEFAULT_TABLE table=test-table [id=5e641ebbf7bd433e819b78f00bc7ce4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:00.202924 11774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f959d3dc29465289a18a5dc65be02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:00.204489 12027 ts_tablet_manager.cc:1905] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:00.206709 12002 tablet_bootstrap.cc:492] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346: Bootstrap starting.
I20260927 15:13:00.208231 12002 tablet_bootstrap.cc:654] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:00.209228 12003 tablet_bootstrap.cc:492] T f6f959d3dc29465289a18a5dc65be02a P 793d49b1ae01412aaa22f1172a2854a8: Bootstrap starting.
I20260927 15:13:00.202076 11912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f959d3dc29465289a18a5dc65be02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:00.210754 12003 tablet_bootstrap.cc:654] T f6f959d3dc29465289a18a5dc65be02a P 793d49b1ae01412aaa22f1172a2854a8: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:00.216760 12006 tablet_bootstrap.cc:492] T f6f959d3dc29465289a18a5dc65be02a P 2b0da842519a49f881bf9c43ea34941f: Bootstrap starting.
I20260927 15:13:00.218523 12006 tablet_bootstrap.cc:654] T f6f959d3dc29465289a18a5dc65be02a P 2b0da842519a49f881bf9c43ea34941f: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:00.224179 12028 ts_tablet_manager.cc:1905] T 21ac147ec412413c8bfe10ee1abfcd6e P 793d49b1ae01412aaa22f1172a2854a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:00.225768 12027 ts_tablet_manager.cc:1918] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:13:00.228338 12026 tablet_replica.cc:330] T 21ac147ec412413c8bfe10ee1abfcd6e P e9b74eb460c448d4865dd19e74ffc346: stopping tablet replica
I20260927 15:13:00.228820 12026 raft_consensus.cc:2229] T 21ac147ec412413c8bfe10ee1abfcd6e P e9b74eb460c448d4865dd19e74ffc346 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:00.229331 12026 raft_consensus.cc:2258] T 21ac147ec412413c8bfe10ee1abfcd6e P e9b74eb460c448d4865dd19e74ffc346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:00.230995 12006 tablet_bootstrap.cc:492] T f6f959d3dc29465289a18a5dc65be02a P 2b0da842519a49f881bf9c43ea34941f: No bootstrap required, opened a new log
I20260927 15:13:00.231101 12027 log.cc:1196] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/21ac147ec412413c8bfe10ee1abfcd6e
I20260927 15:13:00.231223 12006 ts_tablet_manager.cc:1397] T f6f959d3dc29465289a18a5dc65be02a P 2b0da842519a49f881bf9c43ea34941f: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.009s
I20260927 15:13:00.231688 12027 ts_tablet_manager.cc:1939] T 21ac147ec412413c8bfe10ee1abfcd6e P 2b0da842519a49f881bf9c43ea34941f: Deleting consensus metadata
I20260927 15:13:00.232117 12006 raft_consensus.cc:348] T f6f959d3dc29465289a18a5dc65be02a P 2b0da842519a49f881bf9c43ea34941f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:13:00.232357 12006 raft_consensus.cc:374] T f6f959d3dc29465289a18a5dc65be02a P 2b0da842519a49f881bf9c43ea34941f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:00.232450 12006 raft_consensus.cc:729] T f6f959d3dc29465289a18a5dc65be02a P 2b0da842519a49f881bf9c43ea34941f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0da842519a49f881bf9c43ea34941f, State: Initialized, Role: FOLLOWER
I20260927 15:13:00.234754 12006 consensus_queue.cc:260] T f6f959d3dc29465289a18a5dc65be02a P 2b0da842519a49f881bf9c43ea34941f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:13:00.235292 12003 tablet_bootstrap.cc:492] T f6f959d3dc29465289a18a5dc65be02a P 793d49b1ae01412aaa22f1172a2854a8: No bootstrap required, opened a new log
I20260927 15:13:00.235455 12003 ts_tablet_manager.cc:1397] T f6f959d3dc29465289a18a5dc65be02a P 793d49b1ae01412aaa22f1172a2854a8: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.007s
I20260927 15:13:00.236340 12026 ts_tablet_manager.cc:1905] T 21ac147ec412413c8bfe10ee1abfcd6e P e9b74eb460c448d4865dd19e74ffc346: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:00.236431 12002 tablet_bootstrap.cc:492] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346: No bootstrap required, opened a new log
I20260927 15:13:00.236579 12002 ts_tablet_manager.cc:1397] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.011s
I20260927 15:13:00.237491 11483 catalog_manager.cc:4841] TS 2b0da842519a49f881bf9c43ea34941f (127.10.33.67:43983): tablet 21ac147ec412413c8bfe10ee1abfcd6e (table test-table [id=c48bf2b642f948ef83759b4cecdeb558]) successfully deleted
I20260927 15:13:00.238104 12006 ts_tablet_manager.cc:1428] T f6f959d3dc29465289a18a5dc65be02a P 2b0da842519a49f881bf9c43ea34941f: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20260927 15:13:00.238323 12003 raft_consensus.cc:348] T f6f959d3dc29465289a18a5dc65be02a P 793d49b1ae01412aaa22f1172a2854a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:13:00.238555 12003 raft_consensus.cc:374] T f6f959d3dc29465289a18a5dc65be02a P 793d49b1ae01412aaa22f1172a2854a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:00.238657 12003 raft_consensus.cc:729] T f6f959d3dc29465289a18a5dc65be02a P 793d49b1ae01412aaa22f1172a2854a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793d49b1ae01412aaa22f1172a2854a8, State: Initialized, Role: FOLLOWER
I20260927 15:13:00.238871 12002 raft_consensus.cc:348] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:13:00.239089 12002 raft_consensus.cc:374] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:00.239187 12002 raft_consensus.cc:729] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b74eb460c448d4865dd19e74ffc346, State: Initialized, Role: FOLLOWER
I20260927 15:13:00.239452 12002 consensus_queue.cc:260] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:13:00.239831 12003 consensus_queue.cc:260] T f6f959d3dc29465289a18a5dc65be02a P 793d49b1ae01412aaa22f1172a2854a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:13:00.242440 12002 ts_tablet_manager.cc:1428] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20260927 15:13:00.243011 12003 ts_tablet_manager.cc:1428] T f6f959d3dc29465289a18a5dc65be02a P 793d49b1ae01412aaa22f1172a2854a8: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260927 15:13:00.245719 12028 ts_tablet_manager.cc:1918] T 21ac147ec412413c8bfe10ee1abfcd6e P 793d49b1ae01412aaa22f1172a2854a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:13:00.245944 12028 log.cc:1196] T 21ac147ec412413c8bfe10ee1abfcd6e P 793d49b1ae01412aaa22f1172a2854a8: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/21ac147ec412413c8bfe10ee1abfcd6e
I20260927 15:13:00.246495 12028 ts_tablet_manager.cc:1939] T 21ac147ec412413c8bfe10ee1abfcd6e P 793d49b1ae01412aaa22f1172a2854a8: Deleting consensus metadata
I20260927 15:13:00.248483 11483 catalog_manager.cc:4841] TS 793d49b1ae01412aaa22f1172a2854a8 (127.10.33.66:36703): tablet 21ac147ec412413c8bfe10ee1abfcd6e (table test-table [id=c48bf2b642f948ef83759b4cecdeb558]) successfully deleted
I20260927 15:13:00.250146 12026 ts_tablet_manager.cc:1918] T 21ac147ec412413c8bfe10ee1abfcd6e P e9b74eb460c448d4865dd19e74ffc346: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:13:00.250315 12026 log.cc:1196] T 21ac147ec412413c8bfe10ee1abfcd6e P e9b74eb460c448d4865dd19e74ffc346: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/21ac147ec412413c8bfe10ee1abfcd6e
I20260927 15:13:00.250842 12026 ts_tablet_manager.cc:1939] T 21ac147ec412413c8bfe10ee1abfcd6e P e9b74eb460c448d4865dd19e74ffc346: Deleting consensus metadata
I20260927 15:13:00.252614 11483 catalog_manager.cc:4841] TS e9b74eb460c448d4865dd19e74ffc346 (127.10.33.65:41025): tablet 21ac147ec412413c8bfe10ee1abfcd6e (table test-table [id=c48bf2b642f948ef83759b4cecdeb558]) successfully deleted
I20260927 15:13:00.343910 12008 raft_consensus.cc:482] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:13:00.344210 12008 raft_consensus.cc:504] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:13:00.346143 12008 leader_election.cc:288] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793d49b1ae01412aaa22f1172a2854a8 (127.10.33.66:36703), 2b0da842519a49f881bf9c43ea34941f (127.10.33.67:43983)
I20260927 15:13:00.353178 11999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:00.346598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:45605 (local address 127.10.33.66:36703)
0927 15:13:00.346791 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:00.346798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:00.347013 (+ 215us) server_negotiation.cc:407] Connection header received
0927 15:13:00.347231 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:00.347237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:00.347310 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:00.347374 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:00.348243 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.349184 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.351284 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.352078 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.352226 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:00.352300 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:00.352731 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 15:13:00.352872 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:13:00.354791 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:00.346416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:36703 (local address 127.10.33.65:45605)
0927 15:13:00.346897 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:00.346914 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:00.347064 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:00.347592 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:00.347597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:00.347637 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:00.348092 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:00.348105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.349350 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.349356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:00.351131 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:00.351143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.352297 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.352303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:00.352527 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:00.352564 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 15:13:00.353448 (+ 884us) client_negotiation.cc:240] Negotiation successful
0927 15:13:00.353592 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":31,"thread_start_us":380,"threads_started":1}
I20260927 15:13:00.355495 11795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f959d3dc29465289a18a5dc65be02a" candidate_uuid: "e9b74eb460c448d4865dd19e74ffc346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793d49b1ae01412aaa22f1172a2854a8" is_pre_election: true
I20260927 15:13:00.355749 11795 raft_consensus.cc:2454] T f6f959d3dc29465289a18a5dc65be02a P 793d49b1ae01412aaa22f1172a2854a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9b74eb460c448d4865dd19e74ffc346 in term 0.
I20260927 15:13:00.355808 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:00.347205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:43983 (local address 127.10.33.65:50761)
0927 15:13:00.348205 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:00.348223 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:00.348355 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:00.348723 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:00.348728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:00.348753 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:00.349085 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:00.349098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.350490 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.350495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:00.352612 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:00.352627 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.354580 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.354588 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:00.354787 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:00.354821 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 15:13:00.355167 (+ 346us) client_negotiation.cc:240] Negotiation successful
0927 15:13:00.355315 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":3,"thread_start_us":624,"threads_started":1}
I20260927 15:13:00.356271 11586 leader_election.cc:302] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793d49b1ae01412aaa22f1172a2854a8, e9b74eb460c448d4865dd19e74ffc346; no voters:
I20260927 15:13:00.356415 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:00.347466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:50761 (local address 127.10.33.67:43983)
0927 15:13:00.347641 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:00.347647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:00.348321 (+ 674us) server_negotiation.cc:407] Connection header received
0927 15:13:00.348490 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:00.348495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:00.348588 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:00.348695 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:00.349261 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.350359 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.352770 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.354160 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.354315 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:00.354413 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:00.355195 (+ 782us) server_negotiation.cc:299] Negotiation successful
0927 15:13:00.356135 (+ 940us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:13:00.356745 12008 raft_consensus.cc:2781] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:13:00.356860 12008 raft_consensus.cc:482] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:13:00.356938 12008 raft_consensus.cc:3037] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:00.357190 11933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f959d3dc29465289a18a5dc65be02a" candidate_uuid: "e9b74eb460c448d4865dd19e74ffc346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0da842519a49f881bf9c43ea34941f" is_pre_election: true
I20260927 15:13:00.357583 11933 raft_consensus.cc:2454] T f6f959d3dc29465289a18a5dc65be02a P 2b0da842519a49f881bf9c43ea34941f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9b74eb460c448d4865dd19e74ffc346 in term 0.
I20260927 15:13:00.359292 12008 raft_consensus.cc:504] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:13:00.359899 12008 leader_election.cc:288] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346 [CANDIDATE]: Term 1 election: Requested vote from peers 793d49b1ae01412aaa22f1172a2854a8 (127.10.33.66:36703), 2b0da842519a49f881bf9c43ea34941f (127.10.33.67:43983)
I20260927 15:13:00.360433 11795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f959d3dc29465289a18a5dc65be02a" candidate_uuid: "e9b74eb460c448d4865dd19e74ffc346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793d49b1ae01412aaa22f1172a2854a8"
I20260927 15:13:00.360510 11933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f959d3dc29465289a18a5dc65be02a" candidate_uuid: "e9b74eb460c448d4865dd19e74ffc346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0da842519a49f881bf9c43ea34941f"
I20260927 15:13:00.360723 11933 raft_consensus.cc:3037] T f6f959d3dc29465289a18a5dc65be02a P 2b0da842519a49f881bf9c43ea34941f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:00.360723 11795 raft_consensus.cc:3037] T f6f959d3dc29465289a18a5dc65be02a P 793d49b1ae01412aaa22f1172a2854a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:00.362888 11795 raft_consensus.cc:2454] T f6f959d3dc29465289a18a5dc65be02a P 793d49b1ae01412aaa22f1172a2854a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9b74eb460c448d4865dd19e74ffc346 in term 1.
I20260927 15:13:00.363390 11586 leader_election.cc:302] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793d49b1ae01412aaa22f1172a2854a8, e9b74eb460c448d4865dd19e74ffc346; no voters:
I20260927 15:13:00.363685 12008 raft_consensus.cc:2781] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:00.363976 11933 raft_consensus.cc:2454] T f6f959d3dc29465289a18a5dc65be02a P 2b0da842519a49f881bf9c43ea34941f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9b74eb460c448d4865dd19e74ffc346 in term 1.
I20260927 15:13:00.364729 12008 raft_consensus.cc:686] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346 [term 1 LEADER]: Becoming Leader. State: Replica: e9b74eb460c448d4865dd19e74ffc346, State: Running, Role: LEADER
I20260927 15:13:00.365118 12008 consensus_queue.cc:237] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:13:00.369235 11497 catalog_manager.cc:5495] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9b74eb460c448d4865dd19e74ffc346 (127.10.33.65). New cstate: current_term: 1 leader_uuid: "e9b74eb460c448d4865dd19e74ffc346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } health_report { overall_health: UNKNOWN } } }
Sep 27 15:13:00 dist-test-slave-f5h6 krb5kdc[11453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521978, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:00.430651 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:00.416181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:44749 (local address 127.0.0.1:54728)
0927 15:13:00.416841 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:00.416866 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:00.417044 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:00.417425 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:00.417430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:00.418181 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:00.418425 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:00.418434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.419483 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.419487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:00.420431 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:00.420443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.421109 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.421115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:00.421281 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:00.422078 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:00.422096 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:00.424562 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:00.427995 (+ 3433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:00.428038 (+ 43us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:00.428044 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:00.428517 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:00.428861 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:00.428866 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:00.428869 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:00.429064 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:00.429605 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:00.429613 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:00.429777 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 15:13:00.429951 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 15:13:00.430236 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":347,"threads_started":1}
I20260927 15:13:00.430727 12042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:00.416251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54728 (local address 127.10.33.126:44749)
0927 15:13:00.417165 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:00.417170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:00.417189 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:13:00.417228 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:00.417231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:00.417282 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:00.417366 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:00.418579 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.419356 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.420563 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.420955 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.421058 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:00.424778 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:00.424783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:00.424786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:00.424817 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:00.427826 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:00.428631 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:00.428635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:00.428639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:00.428758 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:00.429168 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:00.429173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:00.429176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:00.429493 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:00.429618 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:00.430043 (+ 425us) server_negotiation.cc:299] Negotiation successful
0927 15:13:00.430366 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":796,"thread_start_us":650,"threads_started":1}
I20260927 15:13:00.436671 11497 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:54728:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 15:13:00.437726 11497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:13:00.444893 11635 tablet_service.cc:1467] Processing CreateTablet for tablet 30c2c4c2e7564f60bd9751495a5e2780 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=759624c102d343309a433900ec110632]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:13:00.445490 11635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30c2c4c2e7564f60bd9751495a5e2780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:00.452934 12002 tablet_bootstrap.cc:492] T 30c2c4c2e7564f60bd9751495a5e2780 P e9b74eb460c448d4865dd19e74ffc346: Bootstrap starting.
I20260927 15:13:00.454131 11912 tablet_service.cc:1467] Processing CreateTablet for tablet 30c2c4c2e7564f60bd9751495a5e2780 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=759624c102d343309a433900ec110632]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:13:00.454681 12002 tablet_bootstrap.cc:654] T 30c2c4c2e7564f60bd9751495a5e2780 P e9b74eb460c448d4865dd19e74ffc346: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:00.454691 11912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30c2c4c2e7564f60bd9751495a5e2780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:00.456494 12002 tablet_bootstrap.cc:492] T 30c2c4c2e7564f60bd9751495a5e2780 P e9b74eb460c448d4865dd19e74ffc346: No bootstrap required, opened a new log
I20260927 15:13:00.456645 12002 ts_tablet_manager.cc:1397] T 30c2c4c2e7564f60bd9751495a5e2780 P e9b74eb460c448d4865dd19e74ffc346: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260927 15:13:00.457204 12002 raft_consensus.cc:348] T 30c2c4c2e7564f60bd9751495a5e2780 P e9b74eb460c448d4865dd19e74ffc346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:13:00.457430 12002 raft_consensus.cc:374] T 30c2c4c2e7564f60bd9751495a5e2780 P e9b74eb460c448d4865dd19e74ffc346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:00.457502 12002 raft_consensus.cc:729] T 30c2c4c2e7564f60bd9751495a5e2780 P e9b74eb460c448d4865dd19e74ffc346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b74eb460c448d4865dd19e74ffc346, State: Initialized, Role: FOLLOWER
I20260927 15:13:00.457782 12002 consensus_queue.cc:260] T 30c2c4c2e7564f60bd9751495a5e2780 P e9b74eb460c448d4865dd19e74ffc346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:13:00.458274 12008 tablet_replica.cc:439] T 30c2c4c2e7564f60bd9751495a5e2780 P e9b74eb460c448d4865dd19e74ffc346: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } } }
I20260927 15:13:00.458460 12002 ts_tablet_manager.cc:1428] T 30c2c4c2e7564f60bd9751495a5e2780 P e9b74eb460c448d4865dd19e74ffc346: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:13:00.458590 12008 tablet_replica.cc:442] T 30c2c4c2e7564f60bd9751495a5e2780 P e9b74eb460c448d4865dd19e74ffc346: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:13:00.459554 11774 tablet_service.cc:1467] Processing CreateTablet for tablet 30c2c4c2e7564f60bd9751495a5e2780 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=759624c102d343309a433900ec110632]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:13:00.460158 11774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30c2c4c2e7564f60bd9751495a5e2780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:00.467651 12003 tablet_bootstrap.cc:492] T 30c2c4c2e7564f60bd9751495a5e2780 P 793d49b1ae01412aaa22f1172a2854a8: Bootstrap starting.
I20260927 15:13:00.469360 12003 tablet_bootstrap.cc:654] T 30c2c4c2e7564f60bd9751495a5e2780 P 793d49b1ae01412aaa22f1172a2854a8: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:00.469501 12006 tablet_bootstrap.cc:492] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f: Bootstrap starting.
I20260927 15:13:00.471423 12006 tablet_bootstrap.cc:654] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:00.474256 12006 tablet_bootstrap.cc:492] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f: No bootstrap required, opened a new log
I20260927 15:13:00.474469 12006 ts_tablet_manager.cc:1397] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 15:13:00.475266 12006 raft_consensus.cc:348] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:13:00.475541 12006 raft_consensus.cc:374] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:00.475632 12006 raft_consensus.cc:729] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0da842519a49f881bf9c43ea34941f, State: Initialized, Role: FOLLOWER
I20260927 15:13:00.475884 12006 consensus_queue.cc:260] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:13:00.476313 12012 tablet_replica.cc:439] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } } }
I20260927 15:13:00.476631 12012 tablet_replica.cc:442] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:13:00.477743 12006 ts_tablet_manager.cc:1428] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 15:13:00.484108 12003 tablet_bootstrap.cc:492] T 30c2c4c2e7564f60bd9751495a5e2780 P 793d49b1ae01412aaa22f1172a2854a8: No bootstrap required, opened a new log
I20260927 15:13:00.484338 12003 ts_tablet_manager.cc:1397] T 30c2c4c2e7564f60bd9751495a5e2780 P 793d49b1ae01412aaa22f1172a2854a8: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.011s
I20260927 15:13:00.485024 12003 raft_consensus.cc:348] T 30c2c4c2e7564f60bd9751495a5e2780 P 793d49b1ae01412aaa22f1172a2854a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:13:00.485260 12003 raft_consensus.cc:374] T 30c2c4c2e7564f60bd9751495a5e2780 P 793d49b1ae01412aaa22f1172a2854a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:00.485359 12003 raft_consensus.cc:729] T 30c2c4c2e7564f60bd9751495a5e2780 P 793d49b1ae01412aaa22f1172a2854a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793d49b1ae01412aaa22f1172a2854a8, State: Initialized, Role: FOLLOWER
I20260927 15:13:00.485668 12003 consensus_queue.cc:260] T 30c2c4c2e7564f60bd9751495a5e2780 P 793d49b1ae01412aaa22f1172a2854a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:13:00.486351 12003 ts_tablet_manager.cc:1428] T 30c2c4c2e7564f60bd9751495a5e2780 P 793d49b1ae01412aaa22f1172a2854a8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 15:13:00.486202 12007 tablet_replica.cc:439] T 30c2c4c2e7564f60bd9751495a5e2780 P 793d49b1ae01412aaa22f1172a2854a8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } } }
I20260927 15:13:00.486616 12007 tablet_replica.cc:442] T 30c2c4c2e7564f60bd9751495a5e2780 P 793d49b1ae01412aaa22f1172a2854a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:13:00.795995 12012 raft_consensus.cc:482] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:13:00.796195 12012 raft_consensus.cc:504] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:13:00.796857 12012 leader_election.cc:288] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9b74eb460c448d4865dd19e74ffc346 (127.10.33.65:41025), 793d49b1ae01412aaa22f1172a2854a8 (127.10.33.66:36703)
I20260927 15:13:00.797524 11655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c2c4c2e7564f60bd9751495a5e2780" candidate_uuid: "2b0da842519a49f881bf9c43ea34941f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9b74eb460c448d4865dd19e74ffc346" is_pre_election: true
I20260927 15:13:00.797513 11795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c2c4c2e7564f60bd9751495a5e2780" candidate_uuid: "2b0da842519a49f881bf9c43ea34941f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793d49b1ae01412aaa22f1172a2854a8" is_pre_election: true
I20260927 15:13:00.797748 11655 raft_consensus.cc:2454] T 30c2c4c2e7564f60bd9751495a5e2780 P e9b74eb460c448d4865dd19e74ffc346 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b0da842519a49f881bf9c43ea34941f in term 0.
I20260927 15:13:00.797762 11795 raft_consensus.cc:2454] T 30c2c4c2e7564f60bd9751495a5e2780 P 793d49b1ae01412aaa22f1172a2854a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b0da842519a49f881bf9c43ea34941f in term 0.
I20260927 15:13:00.798251 11865 leader_election.cc:302] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0da842519a49f881bf9c43ea34941f, 793d49b1ae01412aaa22f1172a2854a8; no voters:
I20260927 15:13:00.798553 12012 raft_consensus.cc:2781] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:13:00.798655 12012 raft_consensus.cc:482] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:13:00.798754 12012 raft_consensus.cc:3037] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:00.801067 12012 raft_consensus.cc:504] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:13:00.801646 12012 leader_election.cc:288] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f [CANDIDATE]: Term 1 election: Requested vote from peers e9b74eb460c448d4865dd19e74ffc346 (127.10.33.65:41025), 793d49b1ae01412aaa22f1172a2854a8 (127.10.33.66:36703)
I20260927 15:13:00.802255 11795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c2c4c2e7564f60bd9751495a5e2780" candidate_uuid: "2b0da842519a49f881bf9c43ea34941f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793d49b1ae01412aaa22f1172a2854a8"
I20260927 15:13:00.802345 11655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c2c4c2e7564f60bd9751495a5e2780" candidate_uuid: "2b0da842519a49f881bf9c43ea34941f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9b74eb460c448d4865dd19e74ffc346"
I20260927 15:13:00.802515 11795 raft_consensus.cc:3037] T 30c2c4c2e7564f60bd9751495a5e2780 P 793d49b1ae01412aaa22f1172a2854a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:00.802543 11655 raft_consensus.cc:3037] T 30c2c4c2e7564f60bd9751495a5e2780 P e9b74eb460c448d4865dd19e74ffc346 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:00.805375 11795 raft_consensus.cc:2454] T 30c2c4c2e7564f60bd9751495a5e2780 P 793d49b1ae01412aaa22f1172a2854a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0da842519a49f881bf9c43ea34941f in term 1.
I20260927 15:13:00.805392 11655 raft_consensus.cc:2454] T 30c2c4c2e7564f60bd9751495a5e2780 P e9b74eb460c448d4865dd19e74ffc346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0da842519a49f881bf9c43ea34941f in term 1.
I20260927 15:13:00.805992 11865 leader_election.cc:302] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0da842519a49f881bf9c43ea34941f, e9b74eb460c448d4865dd19e74ffc346; no voters:
I20260927 15:13:00.806277 12012 raft_consensus.cc:2781] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:00.806715 12012 raft_consensus.cc:686] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f [term 1 LEADER]: Becoming Leader. State: Replica: 2b0da842519a49f881bf9c43ea34941f, State: Running, Role: LEADER
I20260927 15:13:00.807015 12012 consensus_queue.cc:237] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } }
I20260927 15:13:00.808555 12046 tablet_replica.cc:439] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f: TxnStatusTablet state changed. Reason: New leader 2b0da842519a49f881bf9c43ea34941f. Latest consensus state: current_term: 1 leader_uuid: "2b0da842519a49f881bf9c43ea34941f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } } }
I20260927 15:13:00.808935 12046 tablet_replica.cc:442] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:13:00.810151 11498 catalog_manager.cc:5495] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b0da842519a49f881bf9c43ea34941f (127.10.33.67). New cstate: current_term: 1 leader_uuid: "2b0da842519a49f881bf9c43ea34941f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } health_report { overall_health: HEALTHY } } }
I20260927 15:13:00.812613 12048 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 15:13:00.812883 12048 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 15:13:00.829489 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:00.824154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:43983 (local address 127.0.0.1:54590)
0927 15:13:00.824384 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:00.824402 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:00.824547 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:00.825014 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:00.825019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:00.825032 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:13:00.825202 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:00.825211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.826359 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.826366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:00.827512 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:00.827525 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.828262 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.828268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:00.828464 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:00.828528 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:13:00.828952 (+ 424us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:13:00.828981 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 15:13:00.829055 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 15:13:00.829223 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 15:13:00.829535 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:00.824175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54590 (local address 127.10.33.67:43983)
0927 15:13:00.824334 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:00.824341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:00.824478 (+ 137us) server_negotiation.cc:407] Connection header received
0927 15:13:00.824751 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:00.824755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:00.824821 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:00.824919 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:13:00.825333 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.826221 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.827645 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.828118 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.828228 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:00.828698 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:13:00.828869 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:13:00.828948 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:00.829082 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 15:13:00.829166 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 15:13:00.835244 11655 raft_consensus.cc:1261] T 30c2c4c2e7564f60bd9751495a5e2780 P e9b74eb460c448d4865dd19e74ffc346 [term 1 FOLLOWER]: Refusing update from remote peer 2b0da842519a49f881bf9c43ea34941f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:00.835251 11795 raft_consensus.cc:1261] T 30c2c4c2e7564f60bd9751495a5e2780 P 793d49b1ae01412aaa22f1172a2854a8 [term 1 FOLLOWER]: Refusing update from remote peer 2b0da842519a49f881bf9c43ea34941f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:00.836153 12012 consensus_queue.cc:1035] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:00.836669 12012 consensus_queue.cc:1035] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f [LEADER]: Connected to new peer: Peer: permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:00.841522 12008 tablet_replica.cc:439] T 30c2c4c2e7564f60bd9751495a5e2780 P e9b74eb460c448d4865dd19e74ffc346: TxnStatusTablet state changed. Reason: New leader 2b0da842519a49f881bf9c43ea34941f. Latest consensus state: current_term: 1 leader_uuid: "2b0da842519a49f881bf9c43ea34941f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } } }
I20260927 15:13:00.841843 12008 tablet_replica.cc:442] T 30c2c4c2e7564f60bd9751495a5e2780 P e9b74eb460c448d4865dd19e74ffc346: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:13:00.844928 12051 mvcc.cc:205] Tried to move back new op lower bound from 7333978033491693568 to 7333978033387872256. Current Snapshot: MvccSnapshot[applied={T|T < 7333978033491693568}]
I20260927 15:13:00.845033 12007 tablet_replica.cc:439] T 30c2c4c2e7564f60bd9751495a5e2780 P 793d49b1ae01412aaa22f1172a2854a8: TxnStatusTablet state changed. Reason: New leader 2b0da842519a49f881bf9c43ea34941f. Latest consensus state: current_term: 1 leader_uuid: "2b0da842519a49f881bf9c43ea34941f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } } }
I20260927 15:13:00.845343 12007 tablet_replica.cc:442] T 30c2c4c2e7564f60bd9751495a5e2780 P 793d49b1ae01412aaa22f1172a2854a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:13:00.845610 12052 mvcc.cc:205] Tried to move back new op lower bound from 7333978033491693568 to 7333978033387872256. Current Snapshot: MvccSnapshot[applied={T|T < 7333978033491693568}]
I20260927 15:13:00.846189 12053 mvcc.cc:205] Tried to move back new op lower bound from 7333978033491693568 to 7333978033387872256. Current Snapshot: MvccSnapshot[applied={T|T < 7333978033491693568}]
I20260927 15:13:00.846088 12008 tablet_replica.cc:439] T 30c2c4c2e7564f60bd9751495a5e2780 P e9b74eb460c448d4865dd19e74ffc346: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b0da842519a49f881bf9c43ea34941f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } } }
I20260927 15:13:00.846334 12008 tablet_replica.cc:442] T 30c2c4c2e7564f60bd9751495a5e2780 P e9b74eb460c448d4865dd19e74ffc346: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:13:00.849684 12046 tablet_replica.cc:439] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b0da842519a49f881bf9c43ea34941f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } } }
I20260927 15:13:00.850046 12046 tablet_replica.cc:442] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:13:00.850255 12007 tablet_replica.cc:439] T 30c2c4c2e7564f60bd9751495a5e2780 P 793d49b1ae01412aaa22f1172a2854a8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b0da842519a49f881bf9c43ea34941f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } } }
I20260927 15:13:00.850477 12007 tablet_replica.cc:442] T 30c2c4c2e7564f60bd9751495a5e2780 P 793d49b1ae01412aaa22f1172a2854a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:13:00.850396 12046 tablet_replica.cc:439] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b0da842519a49f881bf9c43ea34941f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b74eb460c448d4865dd19e74ffc346" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41025 } } peers { permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 } } peers { permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 } } }
I20260927 15:13:00.850606 12046 tablet_replica.cc:442] T 30c2c4c2e7564f60bd9751495a5e2780 P 2b0da842519a49f881bf9c43ea34941f: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:13:00.866513 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:00.860131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.10.33.65:41025)
0927 15:13:00.860742 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:00.860748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:00.860768 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:13:00.860870 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:00.860875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:00.860990 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:00.861156 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:13:00.861530 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.862627 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.863945 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.864644 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.864762 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:00.865162 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:13:00.865928 (+ 766us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:13:00.866068 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:00.866116 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 15:13:00.866198 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":392,"threads_started":1}
I20260927 15:13:00.881778 12008 consensus_queue.cc:1035] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0da842519a49f881bf9c43ea34941f" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:00.883062 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:00.876770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:57293 (local address 127.10.33.67:43983)
0927 15:13:00.876942 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:00.876949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:00.877431 (+ 482us) server_negotiation.cc:407] Connection header received
0927 15:13:00.877662 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:00.877668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:00.877774 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:00.877872 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:13:00.878355 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.879319 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.880896 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.881425 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.881624 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:00.882125 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:13:00.882273 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:13:00.882368 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:00.882660 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 15:13:00.882748 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:13:00.883139 12065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:00.876767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:43983 (local address 127.10.33.65:57293)
0927 15:13:00.877323 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:00.877338 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:00.877484 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:00.877903 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:00.877908 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:00.877934 (+ 26us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:13:00.878211 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:00.878221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.879490 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.879496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:00.880756 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:00.880771 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.881658 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.881664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:00.881837 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:00.881932 (+ 95us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:13:00.882536 (+ 604us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:13:00.882549 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 15:13:00.882623 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 15:13:00.882767 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":262,"threads_started":1}
I20260927 15:13:00.890663 12067 consensus_queue.cc:1035] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793d49b1ae01412aaa22f1172a2854a8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:00.930045 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:00.923962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:51363 (local address 127.10.33.65:41025)
0927 15:13:00.924123 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:00.924129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:00.924207 (+ 78us) server_negotiation.cc:407] Connection header received
0927 15:13:00.924392 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:00.924396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:00.924480 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:00.924576 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:13:00.925194 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.926217 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.927946 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.928645 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.928761 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:00.929297 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:13:00.929438 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:13:00.929534 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:00.929659 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 15:13:00.929738 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:13:00.930403 12070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:00.923354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:41025 (local address 127.10.33.67:51363)
0927 15:13:00.924124 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:00.924144 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:00.924272 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:00.924601 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:00.924604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:00.924632 (+ 28us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:13:00.925052 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:00.925062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.926378 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.926385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:00.927801 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:00.927813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:00.928837 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:00.928845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:00.929041 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:00.929124 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:13:00.929540 (+ 416us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:13:00.929551 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 15:13:00.929633 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 15:13:00.929787 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"spinlock_wait_cycles":3328,"thread_start_us":513,"threads_started":1}
I20260927 15:13:00.971215 11497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:00.971486 11497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:00.975400 11497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a9b3feda4204943ad83dc6938a8c8d2: Sending DeleteTablet for 3 replicas of tablet f6f959d3dc29465289a18a5dc65be02a
I20260927 15:13:00.976673 11774 tablet_service.cc:1514] Processing DeleteTablet for tablet f6f959d3dc29465289a18a5dc65be02a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:00 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:33486
I20260927 15:13:00.976706 11912 tablet_service.cc:1514] Processing DeleteTablet for tablet f6f959d3dc29465289a18a5dc65be02a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:00 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:54576
I20260927 15:13:00.978857 11635 tablet_service.cc:1514] Processing DeleteTablet for tablet f6f959d3dc29465289a18a5dc65be02a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:00 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:48856
I20260927 15:13:00.991950 12076 tablet_replica.cc:330] T f6f959d3dc29465289a18a5dc65be02a P 2b0da842519a49f881bf9c43ea34941f: stopping tablet replica
I20260927 15:13:00.992343 12076 raft_consensus.cc:2229] T f6f959d3dc29465289a18a5dc65be02a P 2b0da842519a49f881bf9c43ea34941f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:00.992705 12076 raft_consensus.cc:2258] T f6f959d3dc29465289a18a5dc65be02a P 2b0da842519a49f881bf9c43ea34941f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:00.993839 12076 ts_tablet_manager.cc:1905] T f6f959d3dc29465289a18a5dc65be02a P 2b0da842519a49f881bf9c43ea34941f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:01.001822 12076 ts_tablet_manager.cc:1918] T f6f959d3dc29465289a18a5dc65be02a P 2b0da842519a49f881bf9c43ea34941f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:13:01.002167 12076 log.cc:1196] T f6f959d3dc29465289a18a5dc65be02a P 2b0da842519a49f881bf9c43ea34941f: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/f6f959d3dc29465289a18a5dc65be02a
I20260927 15:13:01.002708 12076 ts_tablet_manager.cc:1939] T f6f959d3dc29465289a18a5dc65be02a P 2b0da842519a49f881bf9c43ea34941f: Deleting consensus metadata
I20260927 15:13:01.003868 12078 tablet_replica.cc:330] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346: stopping tablet replica
I20260927 15:13:01.003991 11483 catalog_manager.cc:4841] TS 2b0da842519a49f881bf9c43ea34941f (127.10.33.67:43983): tablet f6f959d3dc29465289a18a5dc65be02a (table test-table [id=5e641ebbf7bd433e819b78f00bc7ce4f]) successfully deleted
I20260927 15:13:01.004472 12078 raft_consensus.cc:2229] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:13:01.005131 12078 raft_consensus.cc:2258] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:01.005728 12075 tablet_replica.cc:330] T f6f959d3dc29465289a18a5dc65be02a P 793d49b1ae01412aaa22f1172a2854a8: stopping tablet replica
I20260927 15:13:01.006001 12075 raft_consensus.cc:2229] T f6f959d3dc29465289a18a5dc65be02a P 793d49b1ae01412aaa22f1172a2854a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:01.006306 12075 raft_consensus.cc:2258] T f6f959d3dc29465289a18a5dc65be02a P 793d49b1ae01412aaa22f1172a2854a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:01.007500 12075 ts_tablet_manager.cc:1905] T f6f959d3dc29465289a18a5dc65be02a P 793d49b1ae01412aaa22f1172a2854a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:01.008329 12078 ts_tablet_manager.cc:1905] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 15:13:01 dist-test-slave-f5h6 krb5kdc[11453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521979, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.33.65@KRBTEST.COM
I20260927 15:13:01.019840 12078 ts_tablet_manager.cc:1918] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:13:01.020174 12078 log.cc:1196] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/f6f959d3dc29465289a18a5dc65be02a
I20260927 15:13:01.020820 12078 ts_tablet_manager.cc:1939] T f6f959d3dc29465289a18a5dc65be02a P e9b74eb460c448d4865dd19e74ffc346: Deleting consensus metadata
I20260927 15:13:01.022285 11483 catalog_manager.cc:4841] TS e9b74eb460c448d4865dd19e74ffc346 (127.10.33.65:41025): tablet f6f959d3dc29465289a18a5dc65be02a (table test-table [id=5e641ebbf7bd433e819b78f00bc7ce4f]) successfully deleted
I20260927 15:13:01.024223 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:00.985491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48866 (local address 127.10.33.65:41025)
0927 15:13:00.985713 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:00.985720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:01.002638 (+ 16918us) server_negotiation.cc:407] Connection header received
0927 15:13:01.003187 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:01.003194 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:01.003282 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:01.003360 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:01.005749 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:01.006873 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:01.008798 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:01.009589 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:01.009710 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:01.015514 (+ 5804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:01.015537 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:01.015557 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:01.015613 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:01.020186 (+ 4573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:01.021261 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:01.021272 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:01.021275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:01.021444 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:01.021971 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:01.021977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:01.021980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:01.022487 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:01.022816 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:01.023482 (+ 666us) server_negotiation.cc:299] Negotiation successful
0927 15:13:01.023764 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 15:13:01.025198 11700 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48866
I20260927 15:13:01.031136 12075 ts_tablet_manager.cc:1918] T f6f959d3dc29465289a18a5dc65be02a P 793d49b1ae01412aaa22f1172a2854a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:13:01.031478 12075 log.cc:1196] T f6f959d3dc29465289a18a5dc65be02a P 793d49b1ae01412aaa22f1172a2854a8: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/f6f959d3dc29465289a18a5dc65be02a
I20260927 15:13:01.032003 12075 ts_tablet_manager.cc:1939] T f6f959d3dc29465289a18a5dc65be02a P 793d49b1ae01412aaa22f1172a2854a8: Deleting consensus metadata
I20260927 15:13:01.033499 11483 catalog_manager.cc:4841] TS 793d49b1ae01412aaa22f1172a2854a8 (127.10.33.66:36703): tablet f6f959d3dc29465289a18a5dc65be02a (table test-table [id=5e641ebbf7bd433e819b78f00bc7ce4f]) successfully deleted
I20260927 15:13:01.070415 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:01.039362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.10.33.126:44749)
0927 15:13:01.052271 (+ 12909us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:01.052278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:01.053143 (+ 865us) server_negotiation.cc:407] Connection header received
0927 15:13:01.053231 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:01.053237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:01.053325 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:01.053478 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:01.055474 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:01.056987 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:01.059201 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:01.059889 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:01.060070 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:01.063374 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:01.063382 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:01.063387 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:01.063436 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:01.067096 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:01.068081 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:01.068089 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:01.068092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:01.068263 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:01.068694 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:01.068699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:01.068703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:01.069084 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:01.069345 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:01.069673 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 15:13:01.069966 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12729,"thread_start_us":12928,"threads_started":1}
W20260927 15:13:01.071182 11497 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54732
I20260927 15:13:01.073333 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 11569
I20260927 15:13:01.097643 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:01.077449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48872 (local address 127.10.33.65:41025)
0927 15:13:01.077667 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:01.077673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:01.077694 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:13:01.077955 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:01.077962 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:01.078047 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:01.078117 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:01.079867 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:01.083237 (+ 3370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:01.085350 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:01.086074 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:01.086189 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:01.089472 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:01.089480 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:01.089486 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:01.089538 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:01.093804 (+ 4266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:01.095192 (+ 1388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:01.095200 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:01.095204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:01.095406 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:01.095889 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:01.095895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:01.095899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:01.096299 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:01.096616 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:01.096935 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 15:13:01.097188 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 15:13:01.098584 11700 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48872
W20260927 15:13:01.099259 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 11569: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:13:01.099445 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 11569
I20260927 15:13:01.115741 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 11709
Sep 27 15:13:01 dist-test-slave-f5h6 krb5kdc[11453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521979, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.33.66@KRBTEST.COM
I20260927 15:13:01.132881 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:01.116513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33492 (local address 127.10.33.66:36703)
0927 15:13:01.117044 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:01.117049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:01.117069 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:13:01.117132 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:01.117137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:01.117216 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:01.117322 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:01.118507 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:01.119365 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:01.120718 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:01.121145 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:01.121256 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:01.125977 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:01.125995 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:01.126012 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:01.126162 (+ 150us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:01.129988 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:01.130771 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:01.130779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:01.130781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:01.130935 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:01.131325 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:01.131330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:01.131333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:01.131699 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:01.131981 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:01.132260 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 15:13:01.132493 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":274,"threads_started":1}
W20260927 15:13:01.133488 11839 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33492
W20260927 15:13:01.134152 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 11709: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:13:01.134312 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 11709
I20260927 15:13:01.150197 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 11848
Sep 27 15:13:01 dist-test-slave-f5h6 krb5kdc[11453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521979, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.33.67@KRBTEST.COM
I20260927 15:13:01.170547 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:01.151560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54600 (local address 127.10.33.67:43983)
0927 15:13:01.151781 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:01.151787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:01.151809 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:13:01.151878 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:01.151883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:01.152036 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:01.152196 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:01.153406 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:01.154643 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:01.156426 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:01.157247 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:01.157429 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:01.161984 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:01.162008 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:01.162034 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:01.162225 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:01.166561 (+ 4336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:01.167858 (+ 1297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:01.167869 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:01.167873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:01.168089 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:01.168679 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:01.168685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:01.168688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:01.169183 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:01.169577 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:01.169850 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 15:13:01.170132 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 15:13:01.171378 11978 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54600
W20260927 15:13:01.172238 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 11848: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:13:01.172403 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 11848
I20260927 15:13:01.194070 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 11466
I20260927 15:13:01.195415 11538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:01.316133 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 11466
2026-09-27T15:13:01Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4755 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:13:01 dist-test-slave-f5h6 krb5kdc[12088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:13:01 dist-test-slave-f5h6 krb5kdc[12088](info): set up 2 sockets
Sep 27 15:13:01 dist-test-slave-f5h6 krb5kdc[12088](info): commencing operation
krb5kdc: starting...
W20260927 15:13:02.329717 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:13:02 dist-test-slave-f5h6 krb5kdc[12088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:13:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:13:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:13:02.513901 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:45975
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:45975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:02.832939 12101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:02.833248 12101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:02.833357 12101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:02.840816 12101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:13:02.840929 12101 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 15:13:02.840966 12101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:02.841048 12101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:02.841113 12101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:13:02.841197 12101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:13:02.851373 12101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:45975
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:45975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:02.852667 12101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:02.854008 12101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:02.865576 12106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:02.865527 12109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:02.866212 12107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:02.866461 12101 server_base.cc:1034] running on GCE node
I20260927 15:13:02.867188 12101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:02.868554 12101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:02.869787 12101 hybrid_clock.cc:648] HybridClock initialized: now 1790521982869718 us; error 42 us; skew 500 ppm
I20260927 15:13:02.870146 12101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:02 dist-test-slave-f5h6 krb5kdc[12088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521982, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:02.873400 12101 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:13:02.874910 12101 webserver.cc:460] Webserver started at http://127.10.33.126:38377/ using document root <none> and password file <none>
I20260927 15:13:02.875609 12101 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:02.875706 12101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:02.876000 12101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:02.878933 12101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "f21f133ccd8c4bca9ec1645ef4be01d0"
format_stamp: "Formatted at 2026-09-27 15:13:02 on dist-test-slave-f5h6"
server_key: "2e146ab27ab2bc70063548aa0a8c0b29"
server_key_iv: "e88b2f29c0b6389faca4866d5547ddc4"
server_key_version: "encryptionkey@0"
I20260927 15:13:02.879667 12101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "f21f133ccd8c4bca9ec1645ef4be01d0"
format_stamp: "Formatted at 2026-09-27 15:13:02 on dist-test-slave-f5h6"
server_key: "2e146ab27ab2bc70063548aa0a8c0b29"
server_key_iv: "e88b2f29c0b6389faca4866d5547ddc4"
server_key_version: "encryptionkey@0"
I20260927 15:13:02.884908 12101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 15:13:02.888506 12116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:02.890188 12101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:13:02.890383 12101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "f21f133ccd8c4bca9ec1645ef4be01d0"
format_stamp: "Formatted at 2026-09-27 15:13:02 on dist-test-slave-f5h6"
server_key: "2e146ab27ab2bc70063548aa0a8c0b29"
server_key_iv: "e88b2f29c0b6389faca4866d5547ddc4"
server_key_version: "encryptionkey@0"
I20260927 15:13:02.890606 12101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:02.911056 12101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:02.917165 12101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:02.917588 12101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:02.940424 12101 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:45975
I20260927 15:13:02.940436 12177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:45975 every 8 connection(s)
I20260927 15:13:02.941833 12101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:13:02.945966 12178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:02.951867 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 12101
I20260927 15:13:02.952037 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:13:02.952636 10373 external_mini_cluster.cc:1428] Setting key 043e40985098965a2c1f628020a62103
I20260927 15:13:02.957373 12178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0: Bootstrap starting.
I20260927 15:13:02.961068 12178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:02.962334 12178 log.cc:824] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:02.965812 12178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0: No bootstrap required, opened a new log
Sep 27 15:13:02 dist-test-slave-f5h6 krb5kdc[12088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:02.974936 12178 raft_consensus.cc:348] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21f133ccd8c4bca9ec1645ef4be01d0" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45975 } }
I20260927 15:13:02.975391 12178 raft_consensus.cc:374] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:02.975503 12178 raft_consensus.cc:729] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21f133ccd8c4bca9ec1645ef4be01d0, State: Initialized, Role: FOLLOWER
I20260927 15:13:02.976243 12178 consensus_queue.cc:260] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21f133ccd8c4bca9ec1645ef4be01d0" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45975 } }
I20260927 15:13:02.976480 12178 raft_consensus.cc:388] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:13:02.976617 12178 raft_consensus.cc:482] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:13:02.976771 12178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:02.977557 12181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:02.954623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50316 (local address 127.10.33.126:45975)
0927 15:13:02.955484 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:02.955496 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:02.955540 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:13:02.957216 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:02.957240 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:02.957700 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:02.958114 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:02.959918 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:02.961436 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:02.963193 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:02.964156 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:02.964298 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:02.969064 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:02.969074 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:02.969091 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:02.969144 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:02.972710 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:02.973682 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:02.973690 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:02.973697 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:02.973952 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:02.974434 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:02.974439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:02.974442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:02.975080 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:02.975319 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:02.975822 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 15:13:02.976334 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":376,"threads_started":1}
I20260927 15:13:02.979169 12178 raft_consensus.cc:504] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21f133ccd8c4bca9ec1645ef4be01d0" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45975 } }
I20260927 15:13:02.979674 12178 leader_election.cc:302] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f21f133ccd8c4bca9ec1645ef4be01d0; no voters:
I20260927 15:13:02.980250 12178 leader_election.cc:288] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:13:02.980372 12183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:02.981105 12183 raft_consensus.cc:686] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0 [term 1 LEADER]: Becoming Leader. State: Replica: f21f133ccd8c4bca9ec1645ef4be01d0, State: Running, Role: LEADER
I20260927 15:13:02.981709 12183 consensus_queue.cc:237] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21f133ccd8c4bca9ec1645ef4be01d0" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45975 } }
I20260927 15:13:02.982666 12178 sys_catalog.cc:564] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:13:02.984586 12184 sys_catalog.cc:455] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f21f133ccd8c4bca9ec1645ef4be01d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21f133ccd8c4bca9ec1645ef4be01d0" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45975 } } }
I20260927 15:13:02.985005 12184 sys_catalog.cc:458] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:02.986384 12185 sys_catalog.cc:455] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f21f133ccd8c4bca9ec1645ef4be01d0. Latest consensus state: current_term: 1 leader_uuid: "f21f133ccd8c4bca9ec1645ef4be01d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21f133ccd8c4bca9ec1645ef4be01d0" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45975 } } }
I20260927 15:13:02.986708 12185 sys_catalog.cc:458] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:02.987322 12192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:13:02.992390 12192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:13:03.002341 12192 catalog_manager.cc:1347] Generated new cluster ID: bbc31191294942cda3434a61f53a3c6b
I20260927 15:13:03.002449 12192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:13:03.039345 12192 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:13:03.040768 12192 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:13:03.051409 12192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0: Generated new TSK 0
I20260927 15:13:03.052603 12192 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.33.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
I20260927 15:13:03.106424 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:45975
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:03.433678 12204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:03.434005 12204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:03.434145 12204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:03.441556 12204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:03.441668 12204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:03.441776 12204 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 15:13:03.441838 12204 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:13:03.441958 12204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:13:03.452085 12204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.33.126:45975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:03.453411 12204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:03.454742 12204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:03.467772 12212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:03.467887 12209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:03.467864 12210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:03.468570 12204 server_base.cc:1034] running on GCE node
I20260927 15:13:03.469211 12204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:03.470206 12204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:03.471442 12204 hybrid_clock.cc:648] HybridClock initialized: now 1790521983471380 us; error 48 us; skew 500 ppm
I20260927 15:13:03.471771 12204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:03 dist-test-slave-f5h6 krb5kdc[12088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521983, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:03.474851 12204 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:13:03.476349 12204 webserver.cc:460] Webserver started at http://127.10.33.65:34991/ using document root <none> and password file <none>
I20260927 15:13:03.477097 12204 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:03.477197 12204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:03.477463 12204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:03.480597 12204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "5049bfa6fecf4556873b696e965b888a"
format_stamp: "Formatted at 2026-09-27 15:13:03 on dist-test-slave-f5h6"
server_key: "bc4ffaed5e89c55f61580e99fc49955d"
server_key_iv: "6f6608683215dba77df182ea0c81702b"
server_key_version: "encryptionkey@0"
I20260927 15:13:03.481333 12204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "5049bfa6fecf4556873b696e965b888a"
format_stamp: "Formatted at 2026-09-27 15:13:03 on dist-test-slave-f5h6"
server_key: "bc4ffaed5e89c55f61580e99fc49955d"
server_key_iv: "6f6608683215dba77df182ea0c81702b"
server_key_version: "encryptionkey@0"
I20260927 15:13:03.487305 12204 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 15:13:03.491304 12219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:03.493121 12204 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:13:03.493299 12204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "5049bfa6fecf4556873b696e965b888a"
format_stamp: "Formatted at 2026-09-27 15:13:03 on dist-test-slave-f5h6"
server_key: "bc4ffaed5e89c55f61580e99fc49955d"
server_key_iv: "6f6608683215dba77df182ea0c81702b"
server_key_version: "encryptionkey@0"
I20260927 15:13:03.493506 12204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:03.506609 12204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:03.511282 12204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:03.511572 12204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:03.514381 12204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:03.514495 12204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:03.514614 12204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:03.514712 12204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:13:03 dist-test-slave-f5h6 krb5kdc[12088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521983, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:03.538791 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:03.514850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:50101 (local address 127.10.33.126:45975)
0927 15:13:03.515545 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:03.515552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:03.516570 (+ 1018us) server_negotiation.cc:407] Connection header received
0927 15:13:03.519336 (+ 2766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:03.519342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:03.519430 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:03.519566 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:03.522215 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:03.523155 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:03.525452 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:03.525918 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:03.526059 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:03.531160 (+ 5101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:03.531168 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:03.531172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:03.531215 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:03.534643 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:03.535420 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:03.535426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:03.535431 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:03.535586 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:03.536024 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:03.536029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:03.536033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:03.536436 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:03.536624 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:03.538114 (+ 1490us) server_negotiation.cc:299] Negotiation successful
0927 15:13:03.538384 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":455,"threads_started":1}
I20260927 15:13:03.539696 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:03.515499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:45975 (local address 127.10.33.65:50101)
0927 15:13:03.516370 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:03.516423 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:03.518961 (+ 2538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:03.519879 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:03.519892 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:03.520824 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:03.521971 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:03.522000 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:03.523345 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:03.523349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:03.525291 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:03.525304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:03.526193 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:03.526201 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:03.526586 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:03.527482 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:03.527520 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:03.530853 (+ 3333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:03.534798 (+ 3945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:03.534806 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:03.534821 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:03.535278 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:03.535707 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:03.535711 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:03.535713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:03.535917 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:03.536627 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:03.536636 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:03.537128 (+ 492us) client_negotiation.cc:769] Sending connection context
0927 15:13:03.537497 (+ 369us) client_negotiation.cc:240] Negotiation successful
0927 15:13:03.538119 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":327,"threads_started":1}
I20260927 15:13:03.559930 12303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:03.545593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:45975 (local address 127.10.33.65:53107)
0927 15:13:03.546277 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:03.546294 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:03.546418 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:03.546780 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:03.546784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:03.547218 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:03.547557 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:03.547568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:03.549007 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:03.549012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:03.550140 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:03.550153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:03.551253 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:03.551259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:03.551441 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:03.552193 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:03.552244 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:03.553512 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:03.556810 (+ 3298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:03.556816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:03.556820 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:03.557397 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:03.557786 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:03.557794 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:03.557798 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:03.557988 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:03.558707 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:03.558710 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:03.558929 (+ 219us) client_negotiation.cc:769] Sending connection context
0927 15:13:03.559093 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 15:13:03.559490 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":367,"threads_started":1}
I20260927 15:13:03.560963 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:03.545590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:53107 (local address 127.10.33.126:45975)
0927 15:13:03.545792 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:03.545799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:03.546383 (+ 584us) server_negotiation.cc:407] Connection header received
0927 15:13:03.546528 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:03.546533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:03.546624 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:03.546745 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:03.547690 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:03.548841 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:03.550346 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:03.551011 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:03.551166 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:03.553686 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:03.553691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:03.553694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:03.553726 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:03.556626 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:03.557532 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:03.557538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:03.557542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:03.557676 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:03.558143 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:03.558148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:03.558152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:03.558503 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:03.558652 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:03.560362 (+ 1710us) server_negotiation.cc:299] Negotiation successful
0927 15:13:03.560639 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:13:03.561843 12204 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:37067
I20260927 15:13:03.561964 12339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:37067 every 8 connection(s)
I20260927 15:13:03.563392 12204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:13:03.563851 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 12204
I20260927 15:13:03.564049 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:13:03.564805 10373 external_mini_cluster.cc:1428] Setting key 9665d0c774a3ef754b7224b3d663bf77
I20260927 15:13:03.566761 12340 heartbeater.cc:347] Connected to a master server at 127.10.33.126:45975
I20260927 15:13:03.566983 12340 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:03.567744 12340 heartbeater.cc:511] Master 127.10.33.126:45975 requested a full tablet report, sending...
I20260927 15:13:03.569712 12132 ts_manager.cc:194] Registered new tserver with Master: 5049bfa6fecf4556873b696e965b888a (127.10.33.65:37067)
I20260927 15:13:03.571179 12132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:50101
WARNING: no policy specified for kudu/127.10.33.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
I20260927 15:13:03.610657 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:45975
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:03.934650 12344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:03.934933 12344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:03.935071 12344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:03.943925 12344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:03.944042 12344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:03.944175 12344 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 15:13:03.944233 12344 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:13:03.944344 12344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:13:03.954272 12344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.33.126:45975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:03.955549 12344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:03.956856 12344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:03.970078 12350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:03.970120 12352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:03.970041 12349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:03.975720 12344 server_base.cc:1034] running on GCE node
I20260927 15:13:03.976408 12344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:03.977469 12344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:03.978691 12344 hybrid_clock.cc:648] HybridClock initialized: now 1790521983978639 us; error 38 us; skew 500 ppm
I20260927 15:13:03.979054 12344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:03 dist-test-slave-f5h6 krb5kdc[12088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521983, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:03.982159 12344 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:13:03.983724 12344 webserver.cc:460] Webserver started at http://127.10.33.66:43107/ using document root <none> and password file <none>
I20260927 15:13:03.984449 12344 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:03.984550 12344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:03.984833 12344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:03.987996 12344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "0419aec032574ff49eee704aabbf60be"
format_stamp: "Formatted at 2026-09-27 15:13:03 on dist-test-slave-f5h6"
server_key: "ea6c88b0f60e5e40b5d03a03a2424ba4"
server_key_iv: "9f1a4f1aa8d37e1a0e2b9770f12b0e3a"
server_key_version: "encryptionkey@0"
I20260927 15:13:03.988768 12344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "0419aec032574ff49eee704aabbf60be"
format_stamp: "Formatted at 2026-09-27 15:13:03 on dist-test-slave-f5h6"
server_key: "ea6c88b0f60e5e40b5d03a03a2424ba4"
server_key_iv: "9f1a4f1aa8d37e1a0e2b9770f12b0e3a"
server_key_version: "encryptionkey@0"
I20260927 15:13:03.996065 12344 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260927 15:13:04.000711 12359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:04.002471 12344 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:13:04.002661 12344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "0419aec032574ff49eee704aabbf60be"
format_stamp: "Formatted at 2026-09-27 15:13:03 on dist-test-slave-f5h6"
server_key: "ea6c88b0f60e5e40b5d03a03a2424ba4"
server_key_iv: "9f1a4f1aa8d37e1a0e2b9770f12b0e3a"
server_key_version: "encryptionkey@0"
I20260927 15:13:04.002882 12344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:04.017494 12344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:04.021745 12344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:04.022045 12344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:04.024768 12344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:04.024878 12344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:04.024997 12344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:04.025079 12344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:13:04 dist-test-slave-f5h6 krb5kdc[12088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521983, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:04.047809 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:04.025446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:58783 (local address 127.10.33.126:45975)
0927 15:13:04.025614 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:04.025621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:04.027075 (+ 1454us) server_negotiation.cc:407] Connection header received
0927 15:13:04.029436 (+ 2361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:04.029444 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:04.029592 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:04.029708 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:04.031999 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.032967 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.035216 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.035774 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.035896 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:04.040719 (+ 4823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:04.040727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:04.040732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:04.040773 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:04.044089 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:04.044769 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:04.044774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:04.044778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:04.044917 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:04.045526 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:04.045531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:04.045535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:04.045924 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:04.046079 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:04.047197 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0927 15:13:04.047444 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:13:04.049618 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:04.026042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:45975 (local address 127.10.33.66:58783)
0927 15:13:04.026880 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:04.026940 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:04.029046 (+ 2106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:04.030115 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:04.030127 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:04.030885 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:04.031812 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:04.031833 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.033122 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.033127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:04.035064 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:04.035084 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.036225 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.036230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:04.036565 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:04.037414 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:04.037441 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:04.040477 (+ 3036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:04.044212 (+ 3735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:04.044221 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:04.044239 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:04.044643 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:04.045057 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:04.045062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:04.045066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:04.045389 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:04.046102 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:04.046110 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:04.046706 (+ 596us) client_negotiation.cc:769] Sending connection context
0927 15:13:04.047096 (+ 390us) client_negotiation.cc:240] Negotiation successful
0927 15:13:04.047732 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":347,"threads_started":1}
I20260927 15:13:04.062049 12344 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:44521
I20260927 15:13:04.062093 12478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:44521 every 8 connection(s)
I20260927 15:13:04.063723 12344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:13:04.066278 12479 heartbeater.cc:347] Connected to a master server at 127.10.33.126:45975
I20260927 15:13:04.066493 12479 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:04.067195 12479 heartbeater.cc:511] Master 127.10.33.126:45975 requested a full tablet report, sending...
I20260927 15:13:04.068559 12132 ts_manager.cc:194] Registered new tserver with Master: 0419aec032574ff49eee704aabbf60be (127.10.33.66:44521)
I20260927 15:13:04.068742 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 12344
I20260927 15:13:04.068943 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:13:04.069437 12132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:58783
I20260927 15:13:04.069521 10373 external_mini_cluster.cc:1428] Setting key c046a29adc24746a9ffa10298868618e
I20260927 15:13:04.075789 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:04.056269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:38015 (local address 127.10.33.126:45975)
0927 15:13:04.056463 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:04.056470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:04.062875 (+ 6405us) server_negotiation.cc:407] Connection header received
0927 15:13:04.063030 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:04.063034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:04.063121 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:04.063222 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:04.064450 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.065536 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.067152 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.067626 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.067753 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:04.069942 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:04.069950 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:04.069955 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:04.069996 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:04.073070 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:04.073998 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:04.074003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:04.074006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:04.074166 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:04.074551 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:04.074555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:04.074557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:04.074869 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:04.074978 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:04.075382 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 15:13:04.075560 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:13:04.076021 12465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:04.056422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:45975 (local address 127.10.33.66:38015)
0927 15:13:04.062706 (+ 6284us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:04.062727 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:04.062901 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:04.063281 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:04.063285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:04.063855 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:04.064256 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:04.064273 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.065681 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.065689 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:04.067002 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:04.067018 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.067823 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.067831 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:04.068023 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:04.068660 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:04.068686 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:04.069675 (+ 989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:04.073217 (+ 3542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:04.073223 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:04.073228 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:04.073797 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:04.074277 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:04.074281 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:04.074284 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:04.074443 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:04.074971 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:04.074976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:04.075151 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 15:13:04.075285 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 15:13:04.075629 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6125,"thread_start_us":5636,"threads_started":1}
WARNING: no policy specified for kudu/127.10.33.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
I20260927 15:13:04.114086 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:45975
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:04.444340 12483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:04.444654 12483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:04.444788 12483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:04.452469 12483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:04.452636 12483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:04.452773 12483 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 15:13:04.452844 12483 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:13:04.452950 12483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:13:04.463126 12483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.33.126:45975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:04.464567 12483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:04.465920 12483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:04.479123 12488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:04.479144 12489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:04.479056 12491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:04.479873 12483 server_base.cc:1034] running on GCE node
I20260927 15:13:04.480504 12483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:04.481585 12483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:04.482853 12483 hybrid_clock.cc:648] HybridClock initialized: now 1790521984482759 us; error 91 us; skew 500 ppm
I20260927 15:13:04.483217 12483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:04 dist-test-slave-f5h6 krb5kdc[12088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521984, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:04.491408 12483 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:13:04.493064 12483 webserver.cc:460] Webserver started at http://127.10.33.67:33447/ using document root <none> and password file <none>
I20260927 15:13:04.493826 12483 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:04.494064 12483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:04.494442 12483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:04.497583 12483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "ded8f46c8e234b16815e73a471779267"
format_stamp: "Formatted at 2026-09-27 15:13:04 on dist-test-slave-f5h6"
server_key: "6c107963a9302302e08624fa9ad987b3"
server_key_iv: "7d9ac8da7beb4bdf65c25b6c82e82549"
server_key_version: "encryptionkey@0"
I20260927 15:13:04.498435 12483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "ded8f46c8e234b16815e73a471779267"
format_stamp: "Formatted at 2026-09-27 15:13:04 on dist-test-slave-f5h6"
server_key: "6c107963a9302302e08624fa9ad987b3"
server_key_iv: "7d9ac8da7beb4bdf65c25b6c82e82549"
server_key_version: "encryptionkey@0"
I20260927 15:13:04.503983 12483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:13:04.507759 12498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:04.509213 12483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:13:04.509387 12483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "ded8f46c8e234b16815e73a471779267"
format_stamp: "Formatted at 2026-09-27 15:13:04 on dist-test-slave-f5h6"
server_key: "6c107963a9302302e08624fa9ad987b3"
server_key_iv: "7d9ac8da7beb4bdf65c25b6c82e82549"
server_key_version: "encryptionkey@0"
I20260927 15:13:04.509575 12483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:04.545277 12483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:04.549746 12483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:04.550043 12483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:04.552909 12483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:04.553058 12483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:04.553172 12483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:04.553256 12483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:13:04 dist-test-slave-f5h6 krb5kdc[12088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521984, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:04.573908 12340 heartbeater.cc:503] Master 127.10.33.126:45975 was elected leader, sending a full tablet report...
I20260927 15:13:04.578531 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:04.554034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:35871 (local address 127.10.33.126:45975)
0927 15:13:04.554213 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:04.554218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:04.555581 (+ 1363us) server_negotiation.cc:407] Connection header received
0927 15:13:04.557704 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:04.557710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:04.557803 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:04.557932 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:04.560386 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.561606 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.564244 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.565065 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.565232 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:04.570414 (+ 5182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:04.570423 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:04.570428 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:04.570476 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:04.574004 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:04.575221 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:04.575226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:04.575229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:04.575381 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:04.576116 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:04.576121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:04.576124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:04.576780 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:04.576968 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:04.577892 (+ 924us) server_negotiation.cc:299] Negotiation successful
0927 15:13:04.578182 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:13:04.579972 12507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:04.554463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:45975 (local address 127.10.33.67:35871)
0927 15:13:04.555402 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:04.555445 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:04.557376 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:04.558245 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:04.558261 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:04.559128 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:04.560116 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:04.560156 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.561842 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.561848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:04.564030 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:04.564049 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.565521 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.565530 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:04.565903 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:04.566781 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:04.566824 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:04.570038 (+ 3214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:04.574239 (+ 4201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:04.574251 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:04.574278 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:04.574817 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:04.575543 (+ 726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:04.575549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:04.575552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:04.575942 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:04.576917 (+ 975us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:04.576927 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:04.577349 (+ 422us) client_negotiation.cc:769] Sending connection context
0927 15:13:04.577714 (+ 365us) client_negotiation.cc:240] Negotiation successful
0927 15:13:04.578348 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":424,"threads_started":1}
I20260927 15:13:04.603009 12546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:04.586131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:45975 (local address 127.10.33.67:38717)
0927 15:13:04.586880 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:04.586903 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:04.587102 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:04.587480 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:04.587485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:04.588102 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:04.588490 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:04.588505 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.589529 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.589535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:04.591427 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:04.591453 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.592941 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.592949 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:04.593196 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:04.594307 (+ 1111us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:04.594343 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:04.596119 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:04.600133 (+ 4014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:04.600139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:04.600144 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:04.600581 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:04.600988 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:04.600994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:04.600998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:04.601141 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:04.601670 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:04.601675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:04.602104 (+ 429us) client_negotiation.cc:769] Sending connection context
0927 15:13:04.602311 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 15:13:04.602657 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":390,"threads_started":1}
I20260927 15:13:04.603181 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:04.586044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:38717 (local address 127.10.33.126:45975)
0927 15:13:04.586230 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:04.586236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:04.587056 (+ 820us) server_negotiation.cc:407] Connection header received
0927 15:13:04.587238 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:04.587242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:04.587328 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:04.587437 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:04.588680 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.589389 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.591675 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.592682 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.592860 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:04.596464 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:04.596472 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:04.596478 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:04.596527 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:04.599936 (+ 3409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:04.600732 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:04.600739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:04.600742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:04.600875 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:04.601259 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:04.601263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:04.601265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:04.601552 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:04.601685 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:04.602501 (+ 816us) server_negotiation.cc:299] Negotiation successful
0927 15:13:04.602784 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:13:04.618712 12483 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:38171
I20260927 15:13:04.618784 12617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:38171 every 8 connection(s)
I20260927 15:13:04.619987 12483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:13:04.622105 12618 heartbeater.cc:347] Connected to a master server at 127.10.33.126:45975
I20260927 15:13:04.622313 12618 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:04.622994 12618 heartbeater.cc:511] Master 127.10.33.126:45975 requested a full tablet report, sending...
I20260927 15:13:04.624377 12132 ts_manager.cc:194] Registered new tserver with Master: ded8f46c8e234b16815e73a471779267 (127.10.33.67:38171)
I20260927 15:13:04.624645 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 12483
I20260927 15:13:04.624807 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:13:04.625377 10373 external_mini_cluster.cc:1428] Setting key 463a5349831a0928caac0ed0b0f3ad99
I20260927 15:13:04.625540 12132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:35871
I20260927 15:13:04.628234 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:13:04 dist-test-slave-f5h6 krb5kdc[12088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:13:04 dist-test-slave-f5h6 krb5kdc[12088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:04.673405 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:04.659521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50318 (local address 127.10.33.126:45975)
0927 15:13:04.659763 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:04.659769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:04.659994 (+ 225us) server_negotiation.cc:407] Connection header received
0927 15:13:04.660147 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:04.660153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:04.660219 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:04.660303 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:04.661661 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.662439 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.663678 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.664292 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.664404 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:04.668234 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:04.668240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:04.668243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:04.668278 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:04.670898 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:04.671545 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:04.671550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:04.671553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:04.671670 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:04.672022 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:04.672027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:04.672029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:04.672299 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:04.672521 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:04.672812 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 15:13:04.673066 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 15:13:04.677438 12132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:13:04.684415 12132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:13:04.721817 12633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:04.697693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59360 (local address 127.10.33.66:44521)
0927 15:13:04.714978 (+ 17285us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:04.714986 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:04.715009 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:13:04.715077 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:04.715083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:04.715303 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:04.715492 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:04.716079 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.717465 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.719586 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.720712 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.720851 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:04.721077 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:04.721320 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 15:13:04.721535 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17142,"thread_start_us":17039,"threads_started":1}
I20260927 15:13:04.721896 12631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:04.697524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:44521 (local address 127.0.0.1:59360)
0927 15:13:04.712407 (+ 14883us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:04.712425 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:04.712548 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:04.715511 (+ 2963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:04.715516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:04.715542 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:04.715899 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:04.715911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.717695 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.717698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:04.719414 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:04.719435 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.720940 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.720945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:04.721086 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:04.721121 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 15:13:04.721243 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 15:13:04.721449 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14740,"mutex_wait_us":4,"thread_start_us":13773,"threads_started":1}
I20260927 15:13:04.722885 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:04.700082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46190 (local address 127.10.33.65:37067)
0927 15:13:04.713224 (+ 13142us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:04.713234 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:04.714603 (+ 1369us) server_negotiation.cc:407] Connection header received
0927 15:13:04.714705 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:04.714712 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:04.714955 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:04.715141 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:04.715735 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.717174 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.720173 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.721570 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.721704 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:04.721929 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:04.722451 (+ 522us) server_negotiation.cc:299] Negotiation successful
0927 15:13:04.722618 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12965,"thread_start_us":12845,"threads_started":1}
I20260927 15:13:04.723886 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:04.700021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:37067 (local address 127.0.0.1:46190)
0927 15:13:04.714361 (+ 14340us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:04.714382 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:04.714529 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:04.715147 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:04.715152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:04.715176 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:04.715551 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:04.715564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.717304 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.717309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:04.719999 (+ 2690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:04.720010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.722079 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.722085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:04.722260 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:04.722293 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:13:04.722387 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 15:13:04.723515 (+ 1128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14154,"mutex_wait_us":170,"thread_start_us":14013,"threads_started":1}
I20260927 15:13:04.724233 12632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:04.697440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:38171 (local address 127.0.0.1:38224)
0927 15:13:04.710590 (+ 13150us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:04.710724 (+ 134us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:04.710947 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:04.714332 (+ 3385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:04.714344 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:04.714377 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:04.714807 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:04.714820 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.716721 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.716726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:04.719994 (+ 3268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:04.720009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.723137 (+ 3128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.723144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:04.723541 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:04.723571 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 15:13:04.723660 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 15:13:04.723783 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12788,"mutex_wait_us":404,"thread_start_us":15172,"threads_started":1}
I20260927 15:13:04.724807 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:04.697675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38224 (local address 127.10.33.67:38171)
0927 15:13:04.713736 (+ 16061us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:04.713741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:04.713770 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:13:04.713880 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:04.713889 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:04.714112 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:04.714298 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:04.715118 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.716554 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.721199 (+ 4645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.722774 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.722903 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:04.723473 (+ 570us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:04.724177 (+ 704us) server_negotiation.cc:299] Negotiation successful
0927 15:13:04.724501 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15920,"thread_start_us":15737,"threads_started":1}
I20260927 15:13:04.726198 12409 tablet_service.cc:1467] Processing CreateTablet for tablet 41d9bd8c2a024bc9ad136bee63c3dcb2 (DEFAULT_TABLE table=test-table [id=5b9123aa73ee435a938fc45ebec261bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:04.727717 12409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d9bd8c2a024bc9ad136bee63c3dcb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:04.727505 12553 tablet_service.cc:1467] Processing CreateTablet for tablet 41d9bd8c2a024bc9ad136bee63c3dcb2 (DEFAULT_TABLE table=test-table [id=5b9123aa73ee435a938fc45ebec261bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:04.727599 12270 tablet_service.cc:1467] Processing CreateTablet for tablet 41d9bd8c2a024bc9ad136bee63c3dcb2 (DEFAULT_TABLE table=test-table [id=5b9123aa73ee435a938fc45ebec261bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:04.728638 12553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d9bd8c2a024bc9ad136bee63c3dcb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:04.728910 12270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d9bd8c2a024bc9ad136bee63c3dcb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:04.740939 12637 tablet_bootstrap.cc:492] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P ded8f46c8e234b16815e73a471779267: Bootstrap starting.
I20260927 15:13:04.744302 12637 tablet_bootstrap.cc:654] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P ded8f46c8e234b16815e73a471779267: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:04.745158 12638 tablet_bootstrap.cc:492] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 0419aec032574ff49eee704aabbf60be: Bootstrap starting.
I20260927 15:13:04.745584 12637 log.cc:824] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P ded8f46c8e234b16815e73a471779267: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:04.746069 12639 tablet_bootstrap.cc:492] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a: Bootstrap starting.
I20260927 15:13:04.750875 12638 tablet_bootstrap.cc:654] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 0419aec032574ff49eee704aabbf60be: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:04.752082 12638 log.cc:824] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 0419aec032574ff49eee704aabbf60be: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:04.753527 12639 tablet_bootstrap.cc:654] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:04.754536 12639 log.cc:824] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:04.754915 12638 tablet_bootstrap.cc:492] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 0419aec032574ff49eee704aabbf60be: No bootstrap required, opened a new log
I20260927 15:13:04.754916 12637 tablet_bootstrap.cc:492] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P ded8f46c8e234b16815e73a471779267: No bootstrap required, opened a new log
I20260927 15:13:04.755211 12638 ts_tablet_manager.cc:1397] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 0419aec032574ff49eee704aabbf60be: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260927 15:13:04.755227 12637 ts_tablet_manager.cc:1397] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P ded8f46c8e234b16815e73a471779267: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.005s
I20260927 15:13:04.757189 12639 tablet_bootstrap.cc:492] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a: No bootstrap required, opened a new log
I20260927 15:13:04.757462 12639 ts_tablet_manager.cc:1397] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20260927 15:13:04.764851 12639 raft_consensus.cc:348] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } }
I20260927 15:13:04.765260 12639 raft_consensus.cc:374] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:04.765349 12639 raft_consensus.cc:729] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5049bfa6fecf4556873b696e965b888a, State: Initialized, Role: FOLLOWER
I20260927 15:13:04.765736 12637 raft_consensus.cc:348] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } }
I20260927 15:13:04.765736 12638 raft_consensus.cc:348] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } }
I20260927 15:13:04.766026 12639 consensus_queue.cc:260] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } }
I20260927 15:13:04.766204 12637 raft_consensus.cc:374] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:04.766203 12638 raft_consensus.cc:374] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:04.766292 12638 raft_consensus.cc:729] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0419aec032574ff49eee704aabbf60be, State: Initialized, Role: FOLLOWER
I20260927 15:13:04.766292 12637 raft_consensus.cc:729] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded8f46c8e234b16815e73a471779267, State: Initialized, Role: FOLLOWER
I20260927 15:13:04.766913 12638 consensus_queue.cc:260] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 0419aec032574ff49eee704aabbf60be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } }
I20260927 15:13:04.766914 12637 consensus_queue.cc:260] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P ded8f46c8e234b16815e73a471779267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } }
I20260927 15:13:04.768041 12639 ts_tablet_manager.cc:1428] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a: Time spent starting tablet: real 0.010s user 0.004s sys 0.008s
I20260927 15:13:04.769371 12618 heartbeater.cc:503] Master 127.10.33.126:45975 was elected leader, sending a full tablet report...
I20260927 15:13:04.769654 12638 ts_tablet_manager.cc:1428] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 0419aec032574ff49eee704aabbf60be: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260927 15:13:04.769908 12637 ts_tablet_manager.cc:1428] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P ded8f46c8e234b16815e73a471779267: Time spent starting tablet: real 0.015s user 0.009s sys 0.007s
I20260927 15:13:04.770802 12479 heartbeater.cc:503] Master 127.10.33.126:45975 was elected leader, sending a full tablet report...
W20260927 15:13:04.815878 12480 tablet.cc:2369] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 0419aec032574ff49eee704aabbf60be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:13:04.817463 12341 tablet.cc:2369] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:13:04.853603 12643 raft_consensus.cc:482] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:13:04.854008 12643 raft_consensus.cc:504] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } }
I20260927 15:13:04.855795 12643 leader_election.cc:288] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ded8f46c8e234b16815e73a471779267 (127.10.33.67:38171), 0419aec032574ff49eee704aabbf60be (127.10.33.66:44521)
I20260927 15:13:04.864770 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:04.856312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:42081 (local address 127.10.33.67:38171)
0927 15:13:04.856477 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:04.856482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:04.857170 (+ 688us) server_negotiation.cc:407] Connection header received
0927 15:13:04.857384 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:04.857390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:04.857466 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:04.857557 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:04.858149 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.859268 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.862227 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.863514 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.863718 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:04.863796 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:04.864379 (+ 583us) server_negotiation.cc:299] Negotiation successful
0927 15:13:04.864514 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:13:04.864897 12647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:04.856296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:44521 (local address 127.10.33.65:36955)
0927 15:13:04.857423 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:04.857437 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:04.857553 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:04.857896 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:04.857901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:04.857929 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:04.858253 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:04.858262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.859637 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.859642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:04.861982 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:04.862004 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.863823 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.863831 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:04.864127 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:04.864186 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 15:13:04.864295 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 15:13:04.864455 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"mutex_wait_us":172,"thread_start_us":815,"threads_started":1}
I20260927 15:13:04.865195 12633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:04.856229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:36955 (local address 127.10.33.66:44521)
0927 15:13:04.856416 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:04.856423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:04.857531 (+ 1108us) server_negotiation.cc:407] Connection header received
0927 15:13:04.857696 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:04.857702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:04.857762 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:04.857829 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:04.858480 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.859494 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.862235 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.863513 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.863718 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:04.863796 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:04.864723 (+ 927us) server_negotiation.cc:299] Negotiation successful
0927 15:13:04.864842 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:13:04.865445 12646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:04.856282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:38171 (local address 127.10.33.65:42081)
0927 15:13:04.857040 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:04.857056 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:04.857248 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:04.857601 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:04.857606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:04.857629 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:04.857981 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:04.857992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.859414 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.859418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:04.861991 (+ 2573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:04.862004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.863839 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.863844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:04.864120 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:04.864186 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 15:13:04.864992 (+ 806us) client_negotiation.cc:240] Negotiation successful
0927 15:13:04.865137 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":110,"thread_start_us":442,"threads_started":1}
I20260927 15:13:04.866212 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d9bd8c2a024bc9ad136bee63c3dcb2" candidate_uuid: "5049bfa6fecf4556873b696e965b888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0419aec032574ff49eee704aabbf60be" is_pre_election: true
I20260927 15:13:04.866496 12573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d9bd8c2a024bc9ad136bee63c3dcb2" candidate_uuid: "5049bfa6fecf4556873b696e965b888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded8f46c8e234b16815e73a471779267" is_pre_election: true
I20260927 15:13:04.866816 12429 raft_consensus.cc:2454] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5049bfa6fecf4556873b696e965b888a in term 0.
I20260927 15:13:04.866976 12573 raft_consensus.cc:2454] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5049bfa6fecf4556873b696e965b888a in term 0.
I20260927 15:13:04.867503 12223 leader_election.cc:302] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0419aec032574ff49eee704aabbf60be, 5049bfa6fecf4556873b696e965b888a; no voters:
I20260927 15:13:04.867877 12643 raft_consensus.cc:2781] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:13:04.868026 12643 raft_consensus.cc:482] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:13:04.868136 12643 raft_consensus.cc:3037] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:04.870251 12643 raft_consensus.cc:504] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } }
I20260927 15:13:04.870810 12643 leader_election.cc:288] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a [CANDIDATE]: Term 1 election: Requested vote from peers ded8f46c8e234b16815e73a471779267 (127.10.33.67:38171), 0419aec032574ff49eee704aabbf60be (127.10.33.66:44521)
I20260927 15:13:04.871335 12573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d9bd8c2a024bc9ad136bee63c3dcb2" candidate_uuid: "5049bfa6fecf4556873b696e965b888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded8f46c8e234b16815e73a471779267"
I20260927 15:13:04.871330 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d9bd8c2a024bc9ad136bee63c3dcb2" candidate_uuid: "5049bfa6fecf4556873b696e965b888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0419aec032574ff49eee704aabbf60be"
I20260927 15:13:04.871526 12429 raft_consensus.cc:3037] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Advancing to term 1
W20260927 15:13:04.871553 12619 tablet.cc:2369] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P ded8f46c8e234b16815e73a471779267: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:13:04.871600 12573 raft_consensus.cc:3037] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:04.874600 12573 raft_consensus.cc:2454] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P ded8f46c8e234b16815e73a471779267 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5049bfa6fecf4556873b696e965b888a in term 1.
I20260927 15:13:04.874599 12429 raft_consensus.cc:2454] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 0419aec032574ff49eee704aabbf60be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5049bfa6fecf4556873b696e965b888a in term 1.
I20260927 15:13:04.875062 12221 leader_election.cc:302] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0419aec032574ff49eee704aabbf60be, 5049bfa6fecf4556873b696e965b888a, ded8f46c8e234b16815e73a471779267; no voters:
I20260927 15:13:04.875474 12643 raft_consensus.cc:2781] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:04.875937 12643 raft_consensus.cc:686] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a [term 1 LEADER]: Becoming Leader. State: Replica: 5049bfa6fecf4556873b696e965b888a, State: Running, Role: LEADER
I20260927 15:13:04.876438 12643 consensus_queue.cc:237] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } }
I20260927 15:13:04.881502 12132 catalog_manager.cc:5495] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5049bfa6fecf4556873b696e965b888a (127.10.33.65). New cstate: current_term: 1 leader_uuid: "5049bfa6fecf4556873b696e965b888a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } health_report { overall_health: HEALTHY } } }
Sep 27 15:13:04 dist-test-slave-f5h6 krb5kdc[12088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521982, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:04.934523 12655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:04.920414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:45975 (local address 127.0.0.1:50324)
0927 15:13:04.921084 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:04.921105 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:04.921242 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:04.921548 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:04.921551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:04.922388 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:04.922637 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:04.922646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.923691 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.923697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:04.924696 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:04.924708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.925548 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.925554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:04.925683 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:04.926403 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:04.926421 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:04.928689 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:04.931783 (+ 3094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:04.931795 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:04.931801 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:04.932213 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:04.932736 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:04.932740 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:04.932743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:04.932884 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:04.933502 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:04.933509 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:04.933677 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 15:13:04.933837 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 15:13:04.934147 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":333,"threads_started":1}
I20260927 15:13:04.934953 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:04.920383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50324 (local address 127.10.33.126:45975)
0927 15:13:04.920613 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:04.920620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:04.921209 (+ 589us) server_negotiation.cc:407] Connection header received
0927 15:13:04.921333 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:04.921337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:04.921415 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:04.921511 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:04.922776 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.923554 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.924851 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:04.925340 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:04.925448 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:04.928911 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:04.928917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:04.928920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:04.928952 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:04.931617 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:04.932351 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:04.932357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:04.932361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:04.932599 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:04.933025 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:04.933029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:04.933031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:04.933341 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:04.933499 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:04.934018 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 15:13:04.934332 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":97}
I20260927 15:13:04.941226 12132 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:50324:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 15:13:04.942764 12132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:13:04.951427 12553 tablet_service.cc:1467] Processing CreateTablet for tablet 68ace13139b441f9a9606fa7e8ad9056 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e1108f30b9094f6a984cd905b42aa180]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:13:04.951380 12409 tablet_service.cc:1467] Processing CreateTablet for tablet 68ace13139b441f9a9606fa7e8ad9056 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e1108f30b9094f6a984cd905b42aa180]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:13:04.951339 12270 tablet_service.cc:1467] Processing CreateTablet for tablet 68ace13139b441f9a9606fa7e8ad9056 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e1108f30b9094f6a984cd905b42aa180]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:13:04.952507 12409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ace13139b441f9a9606fa7e8ad9056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:04.952518 12553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ace13139b441f9a9606fa7e8ad9056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:04.952577 12270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ace13139b441f9a9606fa7e8ad9056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:04.958912 12637 tablet_bootstrap.cc:492] T 68ace13139b441f9a9606fa7e8ad9056 P ded8f46c8e234b16815e73a471779267: Bootstrap starting.
I20260927 15:13:04.959813 12638 tablet_bootstrap.cc:492] T 68ace13139b441f9a9606fa7e8ad9056 P 0419aec032574ff49eee704aabbf60be: Bootstrap starting.
I20260927 15:13:04.959812 12639 tablet_bootstrap.cc:492] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a: Bootstrap starting.
I20260927 15:13:04.960655 12637 tablet_bootstrap.cc:654] T 68ace13139b441f9a9606fa7e8ad9056 P ded8f46c8e234b16815e73a471779267: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:04.961396 12639 tablet_bootstrap.cc:654] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:04.961398 12638 tablet_bootstrap.cc:654] T 68ace13139b441f9a9606fa7e8ad9056 P 0419aec032574ff49eee704aabbf60be: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:04.972025 12637 tablet_bootstrap.cc:492] T 68ace13139b441f9a9606fa7e8ad9056 P ded8f46c8e234b16815e73a471779267: No bootstrap required, opened a new log
I20260927 15:13:04.972195 12637 ts_tablet_manager.cc:1397] T 68ace13139b441f9a9606fa7e8ad9056 P ded8f46c8e234b16815e73a471779267: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260927 15:13:04.972456 12639 tablet_bootstrap.cc:492] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a: No bootstrap required, opened a new log
I20260927 15:13:04.972630 12639 ts_tablet_manager.cc:1397] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20260927 15:13:04.972913 12637 raft_consensus.cc:348] T 68ace13139b441f9a9606fa7e8ad9056 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:04.973125 12637 raft_consensus.cc:374] T 68ace13139b441f9a9606fa7e8ad9056 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:04.973208 12637 raft_consensus.cc:729] T 68ace13139b441f9a9606fa7e8ad9056 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded8f46c8e234b16815e73a471779267, State: Initialized, Role: FOLLOWER
I20260927 15:13:04.973451 12637 consensus_queue.cc:260] T 68ace13139b441f9a9606fa7e8ad9056 P ded8f46c8e234b16815e73a471779267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:04.973443 12639 raft_consensus.cc:348] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:04.973683 12639 raft_consensus.cc:374] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:04.973775 12639 raft_consensus.cc:729] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5049bfa6fecf4556873b696e965b888a, State: Initialized, Role: FOLLOWER
I20260927 15:13:04.973922 12645 tablet_replica.cc:439] T 68ace13139b441f9a9606fa7e8ad9056 P ded8f46c8e234b16815e73a471779267: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } } }
I20260927 15:13:04.974081 12639 consensus_queue.cc:260] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:04.974284 12645 tablet_replica.cc:442] T 68ace13139b441f9a9606fa7e8ad9056 P ded8f46c8e234b16815e73a471779267: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:13:04.974489 12643 tablet_replica.cc:439] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } } }
I20260927 15:13:04.974740 12643 tablet_replica.cc:442] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:13:04.974740 12637 ts_tablet_manager.cc:1428] T 68ace13139b441f9a9606fa7e8ad9056 P ded8f46c8e234b16815e73a471779267: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 15:13:04.974766 12638 tablet_bootstrap.cc:492] T 68ace13139b441f9a9606fa7e8ad9056 P 0419aec032574ff49eee704aabbf60be: No bootstrap required, opened a new log
I20260927 15:13:04.974900 12638 ts_tablet_manager.cc:1397] T 68ace13139b441f9a9606fa7e8ad9056 P 0419aec032574ff49eee704aabbf60be: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.012s
I20260927 15:13:04.975651 12638 raft_consensus.cc:348] T 68ace13139b441f9a9606fa7e8ad9056 P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:04.975924 12638 raft_consensus.cc:374] T 68ace13139b441f9a9606fa7e8ad9056 P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:04.976033 12638 raft_consensus.cc:729] T 68ace13139b441f9a9606fa7e8ad9056 P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0419aec032574ff49eee704aabbf60be, State: Initialized, Role: FOLLOWER
I20260927 15:13:04.976253 12638 consensus_queue.cc:260] T 68ace13139b441f9a9606fa7e8ad9056 P 0419aec032574ff49eee704aabbf60be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:04.976928 12638 ts_tablet_manager.cc:1428] T 68ace13139b441f9a9606fa7e8ad9056 P 0419aec032574ff49eee704aabbf60be: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 15:13:04.976753 12644 tablet_replica.cc:439] T 68ace13139b441f9a9606fa7e8ad9056 P 0419aec032574ff49eee704aabbf60be: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } } }
I20260927 15:13:04.977161 12644 tablet_replica.cc:442] T 68ace13139b441f9a9606fa7e8ad9056 P 0419aec032574ff49eee704aabbf60be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:13:04.979199 12639 ts_tablet_manager.cc:1428] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260927 15:13:05.037676 12643 raft_consensus.cc:482] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:13:05.037959 12643 raft_consensus.cc:504] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:05.038635 12643 leader_election.cc:288] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0419aec032574ff49eee704aabbf60be (127.10.33.66:44521), ded8f46c8e234b16815e73a471779267 (127.10.33.67:38171)
I20260927 15:13:05.039225 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ace13139b441f9a9606fa7e8ad9056" candidate_uuid: "5049bfa6fecf4556873b696e965b888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0419aec032574ff49eee704aabbf60be" is_pre_election: true
I20260927 15:13:05.039251 12573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ace13139b441f9a9606fa7e8ad9056" candidate_uuid: "5049bfa6fecf4556873b696e965b888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded8f46c8e234b16815e73a471779267" is_pre_election: true
I20260927 15:13:05.039480 12429 raft_consensus.cc:2454] T 68ace13139b441f9a9606fa7e8ad9056 P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5049bfa6fecf4556873b696e965b888a in term 0.
I20260927 15:13:05.039508 12573 raft_consensus.cc:2454] T 68ace13139b441f9a9606fa7e8ad9056 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5049bfa6fecf4556873b696e965b888a in term 0.
I20260927 15:13:05.039891 12221 leader_election.cc:302] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5049bfa6fecf4556873b696e965b888a, ded8f46c8e234b16815e73a471779267; no voters:
I20260927 15:13:05.040235 12643 raft_consensus.cc:2781] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:13:05.040338 12643 raft_consensus.cc:482] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:13:05.040439 12643 raft_consensus.cc:3037] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:05.042605 12643 raft_consensus.cc:504] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:05.043171 12643 leader_election.cc:288] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a [CANDIDATE]: Term 1 election: Requested vote from peers 0419aec032574ff49eee704aabbf60be (127.10.33.66:44521), ded8f46c8e234b16815e73a471779267 (127.10.33.67:38171)
I20260927 15:13:05.043637 12573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ace13139b441f9a9606fa7e8ad9056" candidate_uuid: "5049bfa6fecf4556873b696e965b888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded8f46c8e234b16815e73a471779267"
I20260927 15:13:05.043686 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ace13139b441f9a9606fa7e8ad9056" candidate_uuid: "5049bfa6fecf4556873b696e965b888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0419aec032574ff49eee704aabbf60be"
I20260927 15:13:05.043830 12573 raft_consensus.cc:3037] T 68ace13139b441f9a9606fa7e8ad9056 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:05.043953 12429 raft_consensus.cc:3037] T 68ace13139b441f9a9606fa7e8ad9056 P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:05.045673 12573 raft_consensus.cc:2454] T 68ace13139b441f9a9606fa7e8ad9056 P ded8f46c8e234b16815e73a471779267 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5049bfa6fecf4556873b696e965b888a in term 1.
I20260927 15:13:05.046052 12429 raft_consensus.cc:2454] T 68ace13139b441f9a9606fa7e8ad9056 P 0419aec032574ff49eee704aabbf60be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5049bfa6fecf4556873b696e965b888a in term 1.
I20260927 15:13:05.046097 12221 leader_election.cc:302] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5049bfa6fecf4556873b696e965b888a, ded8f46c8e234b16815e73a471779267; no voters:
I20260927 15:13:05.046562 12643 raft_consensus.cc:2781] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:05.046734 12643 raft_consensus.cc:686] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a [term 1 LEADER]: Becoming Leader. State: Replica: 5049bfa6fecf4556873b696e965b888a, State: Running, Role: LEADER
I20260927 15:13:05.046964 12643 consensus_queue.cc:237] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:05.048468 12648 tablet_replica.cc:439] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a: TxnStatusTablet state changed. Reason: New leader 5049bfa6fecf4556873b696e965b888a. Latest consensus state: current_term: 1 leader_uuid: "5049bfa6fecf4556873b696e965b888a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } } }
I20260927 15:13:05.048765 12648 tablet_replica.cc:442] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:13:05.050048 12133 catalog_manager.cc:5495] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5049bfa6fecf4556873b696e965b888a (127.10.33.65). New cstate: current_term: 1 leader_uuid: "5049bfa6fecf4556873b696e965b888a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } health_report { overall_health: UNKNOWN } } }
I20260927 15:13:05.050833 12660 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 15:13:05.051086 12660 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 15:13:05.086546 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:05.079656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46204 (local address 127.10.33.65:37067)
0927 15:13:05.080396 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:05.080402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:05.080420 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:13:05.080510 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:05.080514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:05.080631 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:05.080768 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:13:05.081195 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:05.082061 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:05.084114 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:05.084593 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:05.084696 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:05.085323 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:13:05.085968 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:13:05.086144 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:05.086214 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 15:13:05.086284 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593}
I20260927 15:13:05.086791 12655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:05.079574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:37067 (local address 127.0.0.1:46204)
0927 15:13:05.079950 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:05.079969 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:05.080102 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:05.080820 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:05.080825 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:05.080839 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:13:05.081041 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:05.081053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:05.082205 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:05.082211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:05.083941 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:05.083958 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:05.084764 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:05.084771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:05.084975 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:05.085088 (+ 113us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:13:05.086078 (+ 990us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:13:05.086104 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:13:05.086197 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 15:13:05.086363 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206}
I20260927 15:13:05.094913 12573 raft_consensus.cc:1261] T 68ace13139b441f9a9606fa7e8ad9056 P ded8f46c8e234b16815e73a471779267 [term 1 FOLLOWER]: Refusing update from remote peer 5049bfa6fecf4556873b696e965b888a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:05.094918 12429 raft_consensus.cc:1261] T 68ace13139b441f9a9606fa7e8ad9056 P 0419aec032574ff49eee704aabbf60be [term 1 FOLLOWER]: Refusing update from remote peer 5049bfa6fecf4556873b696e965b888a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:05.095942 12643 consensus_queue.cc:1035] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:05.096700 12648 consensus_queue.cc:1035] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:05.108103 12644 tablet_replica.cc:439] T 68ace13139b441f9a9606fa7e8ad9056 P 0419aec032574ff49eee704aabbf60be: TxnStatusTablet state changed. Reason: New leader 5049bfa6fecf4556873b696e965b888a. Latest consensus state: current_term: 1 leader_uuid: "5049bfa6fecf4556873b696e965b888a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } } }
I20260927 15:13:05.108397 12644 tablet_replica.cc:442] T 68ace13139b441f9a9606fa7e8ad9056 P 0419aec032574ff49eee704aabbf60be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:13:05.111856 12648 tablet_replica.cc:439] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5049bfa6fecf4556873b696e965b888a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } } }
I20260927 15:13:05.112007 12664 mvcc.cc:205] Tried to move back new op lower bound from 7333978050938327040 to 7333978050754400256. Current Snapshot: MvccSnapshot[applied={T|T < 7333978050938327040}]
I20260927 15:13:05.112128 12648 tablet_replica.cc:442] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:13:05.114454 12644 tablet_replica.cc:439] T 68ace13139b441f9a9606fa7e8ad9056 P 0419aec032574ff49eee704aabbf60be: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5049bfa6fecf4556873b696e965b888a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } } }
I20260927 15:13:05.114677 12644 tablet_replica.cc:442] T 68ace13139b441f9a9606fa7e8ad9056 P 0419aec032574ff49eee704aabbf60be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:13:05.120079 12645 tablet_replica.cc:439] T 68ace13139b441f9a9606fa7e8ad9056 P ded8f46c8e234b16815e73a471779267: TxnStatusTablet state changed. Reason: New leader 5049bfa6fecf4556873b696e965b888a. Latest consensus state: current_term: 1 leader_uuid: "5049bfa6fecf4556873b696e965b888a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } } }
I20260927 15:13:05.120317 12645 tablet_replica.cc:442] T 68ace13139b441f9a9606fa7e8ad9056 P ded8f46c8e234b16815e73a471779267: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:13:05.122315 12643 tablet_replica.cc:439] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5049bfa6fecf4556873b696e965b888a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } } }
I20260927 15:13:05.122977 12643 tablet_replica.cc:442] T 68ace13139b441f9a9606fa7e8ad9056 P 5049bfa6fecf4556873b696e965b888a: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:13:05.126341 12645 tablet_replica.cc:439] T 68ace13139b441f9a9606fa7e8ad9056 P ded8f46c8e234b16815e73a471779267: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5049bfa6fecf4556873b696e965b888a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } } }
I20260927 15:13:05.126585 12645 tablet_replica.cc:442] T 68ace13139b441f9a9606fa7e8ad9056 P ded8f46c8e234b16815e73a471779267: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:13:05.129689 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:05.123262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46218 (local address 127.10.33.65:37067)
0927 15:13:05.124291 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:05.124297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:05.124313 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:13:05.124370 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:05.124374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:05.124449 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:05.124615 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:13:05.125001 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:05.125744 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:05.127587 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:05.128215 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:05.128339 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:05.128873 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:13:05.129035 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:13:05.129125 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:05.129277 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 15:13:05.129363 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":919}
I20260927 15:13:05.145622 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:05.140137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:46525 (local address 127.10.33.65:37067)
0927 15:13:05.140310 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:05.140316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:05.140902 (+ 586us) server_negotiation.cc:407] Connection header received
0927 15:13:05.141107 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:05.141111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:05.141164 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:05.141229 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:13:05.141759 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:05.142570 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:05.143942 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:05.144375 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:05.144481 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:05.144941 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:13:05.145047 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:13:05.145123 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:05.145240 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 15:13:05.145303 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:13:05.145715 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:05.140109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:37067 (local address 127.10.33.65:46525)
0927 15:13:05.140809 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:05.140824 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:05.140964 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:05.141253 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:05.141258 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:05.141268 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:13:05.141638 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:05.141648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:05.142696 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:05.142701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:05.143808 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:05.143820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:05.144542 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:05.144549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:05.144675 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:05.144734 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:13:05.145130 (+ 396us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:13:05.145141 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 15:13:05.145214 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 15:13:05.145340 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":404,"threads_started":1}
I20260927 15:13:05.156427 12429 raft_consensus.cc:1261] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 0419aec032574ff49eee704aabbf60be [term 1 FOLLOWER]: Refusing update from remote peer 5049bfa6fecf4556873b696e965b888a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:05.157083 12573 raft_consensus.cc:1261] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P ded8f46c8e234b16815e73a471779267 [term 1 FOLLOWER]: Refusing update from remote peer 5049bfa6fecf4556873b696e965b888a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:05.156989 12675 consensus_queue.cc:1035] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:05.157577 12648 consensus_queue.cc:1035] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:05.162011 12663 mvcc.cc:205] Tried to move back new op lower bound from 7333978051195068416 to 7333978050057691136. Current Snapshot: MvccSnapshot[applied={T|T < 7333978051195068416}]
I20260927 15:13:05.228084 12131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:05.228370 12131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:05.232002 12131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0: Sending DeleteTablet for 3 replicas of tablet 41d9bd8c2a024bc9ad136bee63c3dcb2
I20260927 15:13:05.233125 12553 tablet_service.cc:1514] Processing DeleteTablet for tablet 41d9bd8c2a024bc9ad136bee63c3dcb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:05 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:38224
I20260927 15:13:05.233207 12269 tablet_service.cc:1514] Processing DeleteTablet for tablet 41d9bd8c2a024bc9ad136bee63c3dcb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:05 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:46190
I20260927 15:13:05.234043 12409 tablet_service.cc:1514] Processing DeleteTablet for tablet 41d9bd8c2a024bc9ad136bee63c3dcb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:05 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:59360
I20260927 15:13:05.234676 12131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:13:05.235424 12131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:13:05.243592 12270 tablet_service.cc:1467] Processing CreateTablet for tablet f67f6aa58dc84429ae24382231f970ce (DEFAULT_TABLE table=test-table [id=87c02d3ddb5342b49c133e9087a2f3ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:05.243772 12552 tablet_service.cc:1467] Processing CreateTablet for tablet f67f6aa58dc84429ae24382231f970ce (DEFAULT_TABLE table=test-table [id=87c02d3ddb5342b49c133e9087a2f3ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:05.244076 12270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67f6aa58dc84429ae24382231f970ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:05.244258 12552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67f6aa58dc84429ae24382231f970ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:05.246872 12408 tablet_service.cc:1467] Processing CreateTablet for tablet f67f6aa58dc84429ae24382231f970ce (DEFAULT_TABLE table=test-table [id=87c02d3ddb5342b49c133e9087a2f3ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:05.247453 12408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67f6aa58dc84429ae24382231f970ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:05.251152 12639 tablet_bootstrap.cc:492] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a: Bootstrap starting.
I20260927 15:13:05.252846 12639 tablet_bootstrap.cc:654] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:05.254730 12638 tablet_bootstrap.cc:492] T f67f6aa58dc84429ae24382231f970ce P 0419aec032574ff49eee704aabbf60be: Bootstrap starting.
I20260927 15:13:05.256605 12638 tablet_bootstrap.cc:654] T f67f6aa58dc84429ae24382231f970ce P 0419aec032574ff49eee704aabbf60be: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:05.258935 12637 tablet_bootstrap.cc:492] T f67f6aa58dc84429ae24382231f970ce P ded8f46c8e234b16815e73a471779267: Bootstrap starting.
I20260927 15:13:05.260746 12637 tablet_bootstrap.cc:654] T f67f6aa58dc84429ae24382231f970ce P ded8f46c8e234b16815e73a471779267: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:05.265051 12637 tablet_bootstrap.cc:492] T f67f6aa58dc84429ae24382231f970ce P ded8f46c8e234b16815e73a471779267: No bootstrap required, opened a new log
I20260927 15:13:05.265317 12637 ts_tablet_manager.cc:1397] T f67f6aa58dc84429ae24382231f970ce P ded8f46c8e234b16815e73a471779267: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260927 15:13:05.266179 12637 raft_consensus.cc:348] T f67f6aa58dc84429ae24382231f970ce P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:05.266588 12637 raft_consensus.cc:374] T f67f6aa58dc84429ae24382231f970ce P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:05.266738 12637 raft_consensus.cc:729] T f67f6aa58dc84429ae24382231f970ce P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded8f46c8e234b16815e73a471779267, State: Initialized, Role: FOLLOWER
I20260927 15:13:05.267132 12637 consensus_queue.cc:260] T f67f6aa58dc84429ae24382231f970ce P ded8f46c8e234b16815e73a471779267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:05.267803 12637 ts_tablet_manager.cc:1428] T f67f6aa58dc84429ae24382231f970ce P ded8f46c8e234b16815e73a471779267: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:13:05.268731 12682 tablet_replica.cc:330] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a: stopping tablet replica
I20260927 15:13:05.270009 12682 raft_consensus.cc:2229] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:13:05.270273 12638 tablet_bootstrap.cc:492] T f67f6aa58dc84429ae24382231f970ce P 0419aec032574ff49eee704aabbf60be: No bootstrap required, opened a new log
I20260927 15:13:05.270473 12638 ts_tablet_manager.cc:1397] T f67f6aa58dc84429ae24382231f970ce P 0419aec032574ff49eee704aabbf60be: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.007s
I20260927 15:13:05.271186 12682 raft_consensus.cc:2258] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:05.271318 12638 raft_consensus.cc:348] T f67f6aa58dc84429ae24382231f970ce P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:05.271588 12638 raft_consensus.cc:374] T f67f6aa58dc84429ae24382231f970ce P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:05.271667 12638 raft_consensus.cc:729] T f67f6aa58dc84429ae24382231f970ce P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0419aec032574ff49eee704aabbf60be, State: Initialized, Role: FOLLOWER
I20260927 15:13:05.272004 12638 consensus_queue.cc:260] T f67f6aa58dc84429ae24382231f970ce P 0419aec032574ff49eee704aabbf60be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:05.273419 12638 ts_tablet_manager.cc:1428] T f67f6aa58dc84429ae24382231f970ce P 0419aec032574ff49eee704aabbf60be: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 15:13:05.276502 12681 tablet_replica.cc:330] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P ded8f46c8e234b16815e73a471779267: stopping tablet replica
I20260927 15:13:05.277640 12681 raft_consensus.cc:2229] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P ded8f46c8e234b16815e73a471779267 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:05.278824 12681 raft_consensus.cc:2258] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P ded8f46c8e234b16815e73a471779267 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:05.280815 12639 tablet_bootstrap.cc:492] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a: No bootstrap required, opened a new log
I20260927 15:13:05.281018 12639 ts_tablet_manager.cc:1397] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.006s
I20260927 15:13:05.281248 12681 ts_tablet_manager.cc:1905] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P ded8f46c8e234b16815e73a471779267: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:05.281826 12639 raft_consensus.cc:348] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:05.282460 12639 raft_consensus.cc:374] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:05.282574 12639 raft_consensus.cc:729] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5049bfa6fecf4556873b696e965b888a, State: Initialized, Role: FOLLOWER
I20260927 15:13:05.282882 12639 consensus_queue.cc:260] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:05.286222 12639 ts_tablet_manager.cc:1428] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260927 15:13:05.288601 12683 tablet_replica.cc:330] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 0419aec032574ff49eee704aabbf60be: stopping tablet replica
I20260927 15:13:05.289057 12683 raft_consensus.cc:2229] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 0419aec032574ff49eee704aabbf60be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:05.289436 12683 raft_consensus.cc:2258] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 0419aec032574ff49eee704aabbf60be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:05.290189 12682 ts_tablet_manager.cc:1905] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:05.291023 12681 ts_tablet_manager.cc:1918] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P ded8f46c8e234b16815e73a471779267: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:13:05.291182 12681 log.cc:1196] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P ded8f46c8e234b16815e73a471779267: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/41d9bd8c2a024bc9ad136bee63c3dcb2
I20260927 15:13:05.291693 12681 ts_tablet_manager.cc:1939] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P ded8f46c8e234b16815e73a471779267: Deleting consensus metadata
I20260927 15:13:05.293557 12118 catalog_manager.cc:4841] TS ded8f46c8e234b16815e73a471779267 (127.10.33.67:38171): tablet 41d9bd8c2a024bc9ad136bee63c3dcb2 (table test-table [id=5b9123aa73ee435a938fc45ebec261bd]) successfully deleted
I20260927 15:13:05.295583 12683 ts_tablet_manager.cc:1905] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 0419aec032574ff49eee704aabbf60be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:05.298300 12682 ts_tablet_manager.cc:1918] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:13:05.298501 12682 log.cc:1196] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/41d9bd8c2a024bc9ad136bee63c3dcb2
I20260927 15:13:05.299028 12682 ts_tablet_manager.cc:1939] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 5049bfa6fecf4556873b696e965b888a: Deleting consensus metadata
I20260927 15:13:05.300925 12119 catalog_manager.cc:4841] TS 5049bfa6fecf4556873b696e965b888a (127.10.33.65:37067): tablet 41d9bd8c2a024bc9ad136bee63c3dcb2 (table test-table [id=5b9123aa73ee435a938fc45ebec261bd]) successfully deleted
I20260927 15:13:05.303028 12683 ts_tablet_manager.cc:1918] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 0419aec032574ff49eee704aabbf60be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:13:05.303172 12683 log.cc:1196] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 0419aec032574ff49eee704aabbf60be: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/41d9bd8c2a024bc9ad136bee63c3dcb2
I20260927 15:13:05.303588 12683 ts_tablet_manager.cc:1939] T 41d9bd8c2a024bc9ad136bee63c3dcb2 P 0419aec032574ff49eee704aabbf60be: Deleting consensus metadata
I20260927 15:13:05.305157 12120 catalog_manager.cc:4841] TS 0419aec032574ff49eee704aabbf60be (127.10.33.66:44521): tablet 41d9bd8c2a024bc9ad136bee63c3dcb2 (table test-table [id=5b9123aa73ee435a938fc45ebec261bd]) successfully deleted
I20260927 15:13:05.364365 12675 raft_consensus.cc:482] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:13:05.364647 12675 raft_consensus.cc:504] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:05.365479 12675 leader_election.cc:288] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0419aec032574ff49eee704aabbf60be (127.10.33.66:44521), ded8f46c8e234b16815e73a471779267 (127.10.33.67:38171)
I20260927 15:13:05.366004 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67f6aa58dc84429ae24382231f970ce" candidate_uuid: "5049bfa6fecf4556873b696e965b888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0419aec032574ff49eee704aabbf60be" is_pre_election: true
I20260927 15:13:05.366089 12573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67f6aa58dc84429ae24382231f970ce" candidate_uuid: "5049bfa6fecf4556873b696e965b888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded8f46c8e234b16815e73a471779267" is_pre_election: true
I20260927 15:13:05.366247 12429 raft_consensus.cc:2454] T f67f6aa58dc84429ae24382231f970ce P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5049bfa6fecf4556873b696e965b888a in term 0.
I20260927 15:13:05.366325 12573 raft_consensus.cc:2454] T f67f6aa58dc84429ae24382231f970ce P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5049bfa6fecf4556873b696e965b888a in term 0.
I20260927 15:13:05.366677 12223 leader_election.cc:302] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0419aec032574ff49eee704aabbf60be, 5049bfa6fecf4556873b696e965b888a; no voters:
I20260927 15:13:05.366982 12675 raft_consensus.cc:2781] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:13:05.367103 12675 raft_consensus.cc:482] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:13:05.367218 12675 raft_consensus.cc:3037] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:05.370277 12675 raft_consensus.cc:504] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:05.370930 12675 leader_election.cc:288] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a [CANDIDATE]: Term 1 election: Requested vote from peers 0419aec032574ff49eee704aabbf60be (127.10.33.66:44521), ded8f46c8e234b16815e73a471779267 (127.10.33.67:38171)
I20260927 15:13:05.371317 12573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67f6aa58dc84429ae24382231f970ce" candidate_uuid: "5049bfa6fecf4556873b696e965b888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded8f46c8e234b16815e73a471779267"
I20260927 15:13:05.371325 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67f6aa58dc84429ae24382231f970ce" candidate_uuid: "5049bfa6fecf4556873b696e965b888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0419aec032574ff49eee704aabbf60be"
I20260927 15:13:05.371498 12573 raft_consensus.cc:3037] T f67f6aa58dc84429ae24382231f970ce P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:05.371511 12429 raft_consensus.cc:3037] T f67f6aa58dc84429ae24382231f970ce P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:05.373615 12573 raft_consensus.cc:2454] T f67f6aa58dc84429ae24382231f970ce P ded8f46c8e234b16815e73a471779267 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5049bfa6fecf4556873b696e965b888a in term 1.
I20260927 15:13:05.373983 12221 leader_election.cc:302] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5049bfa6fecf4556873b696e965b888a, ded8f46c8e234b16815e73a471779267; no voters:
I20260927 15:13:05.374224 12675 raft_consensus.cc:2781] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:05.374300 12429 raft_consensus.cc:2454] T f67f6aa58dc84429ae24382231f970ce P 0419aec032574ff49eee704aabbf60be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5049bfa6fecf4556873b696e965b888a in term 1.
I20260927 15:13:05.374461 12675 raft_consensus.cc:686] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a [term 1 LEADER]: Becoming Leader. State: Replica: 5049bfa6fecf4556873b696e965b888a, State: Running, Role: LEADER
I20260927 15:13:05.374841 12675 consensus_queue.cc:237] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:05.377562 12131 catalog_manager.cc:5495] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5049bfa6fecf4556873b696e965b888a (127.10.33.65). New cstate: current_term: 1 leader_uuid: "5049bfa6fecf4556873b696e965b888a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } health_report { overall_health: UNKNOWN } } }
I20260927 15:13:05.417938 12573 raft_consensus.cc:1261] T f67f6aa58dc84429ae24382231f970ce P ded8f46c8e234b16815e73a471779267 [term 1 FOLLOWER]: Refusing update from remote peer 5049bfa6fecf4556873b696e965b888a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:05.418148 12429 raft_consensus.cc:1261] T f67f6aa58dc84429ae24382231f970ce P 0419aec032574ff49eee704aabbf60be [term 1 FOLLOWER]: Refusing update from remote peer 5049bfa6fecf4556873b696e965b888a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:05.418514 12675 consensus_queue.cc:1035] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:05.418761 12678 consensus_queue.cc:1035] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:13:05.444624 12663 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 15:13:07.045940 12132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:07.046262 12132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:07.051236 12132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0: Sending DeleteTablet for 3 replicas of tablet f67f6aa58dc84429ae24382231f970ce
I20260927 15:13:07.052433 12553 tablet_service.cc:1514] Processing DeleteTablet for tablet f67f6aa58dc84429ae24382231f970ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:07 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:38224
I20260927 15:13:07.052486 12409 tablet_service.cc:1514] Processing DeleteTablet for tablet f67f6aa58dc84429ae24382231f970ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:07 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:59360
I20260927 15:13:07.052438 12269 tablet_service.cc:1514] Processing DeleteTablet for tablet f67f6aa58dc84429ae24382231f970ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:07 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:46190
I20260927 15:13:07.053176 12709 tablet_replica.cc:330] T f67f6aa58dc84429ae24382231f970ce P ded8f46c8e234b16815e73a471779267: stopping tablet replica
I20260927 15:13:07.053366 12709 raft_consensus.cc:2229] T f67f6aa58dc84429ae24382231f970ce P ded8f46c8e234b16815e73a471779267 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:07.053589 12709 raft_consensus.cc:2258] T f67f6aa58dc84429ae24382231f970ce P ded8f46c8e234b16815e73a471779267 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:07.055343 12709 ts_tablet_manager.cc:1905] T f67f6aa58dc84429ae24382231f970ce P ded8f46c8e234b16815e73a471779267: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:07.060530 12709 ts_tablet_manager.cc:1918] T f67f6aa58dc84429ae24382231f970ce P ded8f46c8e234b16815e73a471779267: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:13:07.060657 12709 log.cc:1196] T f67f6aa58dc84429ae24382231f970ce P ded8f46c8e234b16815e73a471779267: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/f67f6aa58dc84429ae24382231f970ce
I20260927 15:13:07.061000 12709 ts_tablet_manager.cc:1939] T f67f6aa58dc84429ae24382231f970ce P ded8f46c8e234b16815e73a471779267: Deleting consensus metadata
I20260927 15:13:07.062059 12118 catalog_manager.cc:4841] TS ded8f46c8e234b16815e73a471779267 (127.10.33.67:38171): tablet f67f6aa58dc84429ae24382231f970ce (table test-table [id=87c02d3ddb5342b49c133e9087a2f3ac]) successfully deleted
I20260927 15:13:07.068038 12710 tablet_replica.cc:330] T f67f6aa58dc84429ae24382231f970ce P 0419aec032574ff49eee704aabbf60be: stopping tablet replica
I20260927 15:13:07.068043 12711 tablet_replica.cc:330] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a: stopping tablet replica
I20260927 15:13:07.068297 12711 raft_consensus.cc:2229] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:13:07.068363 12710 raft_consensus.cc:2229] T f67f6aa58dc84429ae24382231f970ce P 0419aec032574ff49eee704aabbf60be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:07.068780 12710 raft_consensus.cc:2258] T f67f6aa58dc84429ae24382231f970ce P 0419aec032574ff49eee704aabbf60be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:07.068773 12711 raft_consensus.cc:2258] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:07.070859 12710 ts_tablet_manager.cc:1905] T f67f6aa58dc84429ae24382231f970ce P 0419aec032574ff49eee704aabbf60be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:07.073318 12711 ts_tablet_manager.cc:1905] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:07.078719 12710 ts_tablet_manager.cc:1918] T f67f6aa58dc84429ae24382231f970ce P 0419aec032574ff49eee704aabbf60be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:13:07.078945 12710 log.cc:1196] T f67f6aa58dc84429ae24382231f970ce P 0419aec032574ff49eee704aabbf60be: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/f67f6aa58dc84429ae24382231f970ce
I20260927 15:13:07.079442 12710 ts_tablet_manager.cc:1939] T f67f6aa58dc84429ae24382231f970ce P 0419aec032574ff49eee704aabbf60be: Deleting consensus metadata
I20260927 15:13:07.080659 12120 catalog_manager.cc:4841] TS 0419aec032574ff49eee704aabbf60be (127.10.33.66:44521): tablet f67f6aa58dc84429ae24382231f970ce (table test-table [id=87c02d3ddb5342b49c133e9087a2f3ac]) successfully deleted
I20260927 15:13:07.080948 12711 ts_tablet_manager.cc:1918] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:13:07.081132 12711 log.cc:1196] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/f67f6aa58dc84429ae24382231f970ce
I20260927 15:13:07.081575 12711 ts_tablet_manager.cc:1939] T f67f6aa58dc84429ae24382231f970ce P 5049bfa6fecf4556873b696e965b888a: Deleting consensus metadata
I20260927 15:13:07.083074 12119 catalog_manager.cc:4841] TS 5049bfa6fecf4556873b696e965b888a (127.10.33.65:37067): tablet f67f6aa58dc84429ae24382231f970ce (table test-table [id=87c02d3ddb5342b49c133e9087a2f3ac]) successfully deleted
Sep 27 15:13:07 dist-test-slave-f5h6 krb5kdc[12088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521987, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:13:07 dist-test-slave-f5h6 krb5kdc[12088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521987, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:07.119529 12719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:07.103921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52204 (local address 127.10.33.126:45975)
0927 15:13:07.104541 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:07.104547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:07.104575 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:13:07.104766 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:07.104773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:07.104852 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:07.104989 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:07.106494 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:07.107476 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:07.108834 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:07.109303 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:07.109427 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:07.113374 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:07.113381 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:07.113385 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:07.113431 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:07.116667 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:07.117474 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:07.117479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:07.117482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:07.117660 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:07.118051 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:07.118056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:07.118058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:07.118432 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:07.118665 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:07.118980 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 15:13:07.119186 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":286,"threads_started":1}
I20260927 15:13:07.123697 12132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:13:07.124426 12132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:13:07.132437 12409 tablet_service.cc:1467] Processing CreateTablet for tablet 6280122ba0e34c41a231da403f70f861 (DEFAULT_TABLE table=test-table [id=8a8169d57cab4eeda38336ecd086bb2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:07.132406 12269 tablet_service.cc:1467] Processing CreateTablet for tablet 6280122ba0e34c41a231da403f70f861 (DEFAULT_TABLE table=test-table [id=8a8169d57cab4eeda38336ecd086bb2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:07.132455 12553 tablet_service.cc:1467] Processing CreateTablet for tablet 6280122ba0e34c41a231da403f70f861 (DEFAULT_TABLE table=test-table [id=8a8169d57cab4eeda38336ecd086bb2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:07.132916 12553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6280122ba0e34c41a231da403f70f861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:07.132978 12269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6280122ba0e34c41a231da403f70f861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:07.132978 12409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6280122ba0e34c41a231da403f70f861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:07.137565 12720 tablet_bootstrap.cc:492] T 6280122ba0e34c41a231da403f70f861 P ded8f46c8e234b16815e73a471779267: Bootstrap starting.
I20260927 15:13:07.139010 12720 tablet_bootstrap.cc:654] T 6280122ba0e34c41a231da403f70f861 P ded8f46c8e234b16815e73a471779267: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:07.141173 12720 tablet_bootstrap.cc:492] T 6280122ba0e34c41a231da403f70f861 P ded8f46c8e234b16815e73a471779267: No bootstrap required, opened a new log
I20260927 15:13:07.141315 12720 ts_tablet_manager.cc:1397] T 6280122ba0e34c41a231da403f70f861 P ded8f46c8e234b16815e73a471779267: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 15:13:07.141989 12720 raft_consensus.cc:348] T 6280122ba0e34c41a231da403f70f861 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:07.142184 12720 raft_consensus.cc:374] T 6280122ba0e34c41a231da403f70f861 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:07.142266 12720 raft_consensus.cc:729] T 6280122ba0e34c41a231da403f70f861 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded8f46c8e234b16815e73a471779267, State: Initialized, Role: FOLLOWER
I20260927 15:13:07.142479 12720 consensus_queue.cc:260] T 6280122ba0e34c41a231da403f70f861 P ded8f46c8e234b16815e73a471779267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:07.143314 12720 ts_tablet_manager.cc:1428] T 6280122ba0e34c41a231da403f70f861 P ded8f46c8e234b16815e73a471779267: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:13:07.147850 12722 tablet_bootstrap.cc:492] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a: Bootstrap starting.
I20260927 15:13:07.150497 12722 tablet_bootstrap.cc:654] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:07.150883 12723 tablet_bootstrap.cc:492] T 6280122ba0e34c41a231da403f70f861 P 0419aec032574ff49eee704aabbf60be: Bootstrap starting.
I20260927 15:13:07.152621 12723 tablet_bootstrap.cc:654] T 6280122ba0e34c41a231da403f70f861 P 0419aec032574ff49eee704aabbf60be: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:07.153453 12722 tablet_bootstrap.cc:492] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a: No bootstrap required, opened a new log
I20260927 15:13:07.153644 12722 ts_tablet_manager.cc:1397] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:13:07.154455 12722 raft_consensus.cc:348] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:07.154754 12722 raft_consensus.cc:374] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:07.154852 12722 raft_consensus.cc:729] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5049bfa6fecf4556873b696e965b888a, State: Initialized, Role: FOLLOWER
I20260927 15:13:07.155141 12723 tablet_bootstrap.cc:492] T 6280122ba0e34c41a231da403f70f861 P 0419aec032574ff49eee704aabbf60be: No bootstrap required, opened a new log
I20260927 15:13:07.155135 12722 consensus_queue.cc:260] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:07.155290 12723 ts_tablet_manager.cc:1397] T 6280122ba0e34c41a231da403f70f861 P 0419aec032574ff49eee704aabbf60be: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260927 15:13:07.155876 12722 ts_tablet_manager.cc:1428] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:13:07.156185 12723 raft_consensus.cc:348] T 6280122ba0e34c41a231da403f70f861 P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:07.156420 12723 raft_consensus.cc:374] T 6280122ba0e34c41a231da403f70f861 P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:07.156584 12723 raft_consensus.cc:729] T 6280122ba0e34c41a231da403f70f861 P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0419aec032574ff49eee704aabbf60be, State: Initialized, Role: FOLLOWER
I20260927 15:13:07.156837 12723 consensus_queue.cc:260] T 6280122ba0e34c41a231da403f70f861 P 0419aec032574ff49eee704aabbf60be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:07.157629 12723 ts_tablet_manager.cc:1428] T 6280122ba0e34c41a231da403f70f861 P 0419aec032574ff49eee704aabbf60be: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 15:13:07.372277 12692 raft_consensus.cc:482] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:13:07.372478 12692 raft_consensus.cc:504] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:07.373193 12692 leader_election.cc:288] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0419aec032574ff49eee704aabbf60be (127.10.33.66:44521), ded8f46c8e234b16815e73a471779267 (127.10.33.67:38171)
I20260927 15:13:07.374033 12428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6280122ba0e34c41a231da403f70f861" candidate_uuid: "5049bfa6fecf4556873b696e965b888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0419aec032574ff49eee704aabbf60be" is_pre_election: true
I20260927 15:13:07.374033 12573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6280122ba0e34c41a231da403f70f861" candidate_uuid: "5049bfa6fecf4556873b696e965b888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded8f46c8e234b16815e73a471779267" is_pre_election: true
I20260927 15:13:07.374394 12428 raft_consensus.cc:2454] T 6280122ba0e34c41a231da403f70f861 P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5049bfa6fecf4556873b696e965b888a in term 0.
I20260927 15:13:07.374394 12573 raft_consensus.cc:2454] T 6280122ba0e34c41a231da403f70f861 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5049bfa6fecf4556873b696e965b888a in term 0.
I20260927 15:13:07.375043 12223 leader_election.cc:302] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0419aec032574ff49eee704aabbf60be, 5049bfa6fecf4556873b696e965b888a; no voters:
I20260927 15:13:07.375365 12692 raft_consensus.cc:2781] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:13:07.375489 12692 raft_consensus.cc:482] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:13:07.375576 12692 raft_consensus.cc:3037] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:07.377700 12692 raft_consensus.cc:504] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:07.378289 12692 leader_election.cc:288] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a [CANDIDATE]: Term 1 election: Requested vote from peers 0419aec032574ff49eee704aabbf60be (127.10.33.66:44521), ded8f46c8e234b16815e73a471779267 (127.10.33.67:38171)
I20260927 15:13:07.378870 12573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6280122ba0e34c41a231da403f70f861" candidate_uuid: "5049bfa6fecf4556873b696e965b888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded8f46c8e234b16815e73a471779267"
I20260927 15:13:07.378917 12428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6280122ba0e34c41a231da403f70f861" candidate_uuid: "5049bfa6fecf4556873b696e965b888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0419aec032574ff49eee704aabbf60be"
I20260927 15:13:07.379092 12573 raft_consensus.cc:3037] T 6280122ba0e34c41a231da403f70f861 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:07.379139 12428 raft_consensus.cc:3037] T 6280122ba0e34c41a231da403f70f861 P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:07.381094 12573 raft_consensus.cc:2454] T 6280122ba0e34c41a231da403f70f861 P ded8f46c8e234b16815e73a471779267 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5049bfa6fecf4556873b696e965b888a in term 1.
I20260927 15:13:07.381479 12428 raft_consensus.cc:2454] T 6280122ba0e34c41a231da403f70f861 P 0419aec032574ff49eee704aabbf60be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5049bfa6fecf4556873b696e965b888a in term 1.
I20260927 15:13:07.381646 12221 leader_election.cc:302] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5049bfa6fecf4556873b696e965b888a, ded8f46c8e234b16815e73a471779267; no voters:
I20260927 15:13:07.382076 12692 raft_consensus.cc:2781] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:07.382265 12692 raft_consensus.cc:686] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a [term 1 LEADER]: Becoming Leader. State: Replica: 5049bfa6fecf4556873b696e965b888a, State: Running, Role: LEADER
I20260927 15:13:07.382573 12692 consensus_queue.cc:237] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:07.385954 12131 catalog_manager.cc:5495] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5049bfa6fecf4556873b696e965b888a (127.10.33.65). New cstate: current_term: 1 leader_uuid: "5049bfa6fecf4556873b696e965b888a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } health_report { overall_health: UNKNOWN } } }
I20260927 15:13:07.460357 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:07.455106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46568 (local address 127.10.33.65:37067)
0927 15:13:07.455730 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:07.455734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:07.455751 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:13:07.455808 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:07.455812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:07.455870 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:07.455951 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:13:07.456348 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:07.457171 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:07.458481 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:07.458995 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:07.459133 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:07.459615 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:13:07.459723 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:13:07.459797 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:07.459959 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 15:13:07.460035 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":373,"threads_started":1}
I20260927 15:13:07.470585 12573 raft_consensus.cc:1261] T 6280122ba0e34c41a231da403f70f861 P ded8f46c8e234b16815e73a471779267 [term 1 FOLLOWER]: Refusing update from remote peer 5049bfa6fecf4556873b696e965b888a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:07.470685 12428 raft_consensus.cc:1261] T 6280122ba0e34c41a231da403f70f861 P 0419aec032574ff49eee704aabbf60be [term 1 FOLLOWER]: Refusing update from remote peer 5049bfa6fecf4556873b696e965b888a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:07.471254 12697 consensus_queue.cc:1035] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:07.471575 12692 consensus_queue.cc:1035] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:07.520117 12133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:07.520430 12133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:07.524068 12133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0: Sending DeleteTablet for 3 replicas of tablet 6280122ba0e34c41a231da403f70f861
I20260927 15:13:07.525213 12553 tablet_service.cc:1514] Processing DeleteTablet for tablet 6280122ba0e34c41a231da403f70f861 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:07 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:38224
I20260927 15:13:07.525206 12269 tablet_service.cc:1514] Processing DeleteTablet for tablet 6280122ba0e34c41a231da403f70f861 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:07 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:46190
I20260927 15:13:07.525458 12409 tablet_service.cc:1514] Processing DeleteTablet for tablet 6280122ba0e34c41a231da403f70f861 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:07 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:59360
I20260927 15:13:07.525600 12709 tablet_replica.cc:330] T 6280122ba0e34c41a231da403f70f861 P ded8f46c8e234b16815e73a471779267: stopping tablet replica
I20260927 15:13:07.525764 12710 tablet_replica.cc:330] T 6280122ba0e34c41a231da403f70f861 P 0419aec032574ff49eee704aabbf60be: stopping tablet replica
I20260927 15:13:07.525889 12709 raft_consensus.cc:2229] T 6280122ba0e34c41a231da403f70f861 P ded8f46c8e234b16815e73a471779267 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:07.526000 12710 raft_consensus.cc:2229] T 6280122ba0e34c41a231da403f70f861 P 0419aec032574ff49eee704aabbf60be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:07.526090 12709 raft_consensus.cc:2258] T 6280122ba0e34c41a231da403f70f861 P ded8f46c8e234b16815e73a471779267 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:07.526242 12710 raft_consensus.cc:2258] T 6280122ba0e34c41a231da403f70f861 P 0419aec032574ff49eee704aabbf60be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:07.526392 12711 tablet_replica.cc:330] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a: stopping tablet replica
I20260927 15:13:07.526619 12711 raft_consensus.cc:2229] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:13:07.526510 12133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 15:13:07.527021 12711 raft_consensus.cc:2258] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 15:13:07.527268 12133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:13:07.527655 12709 ts_tablet_manager.cc:1905] T 6280122ba0e34c41a231da403f70f861 P ded8f46c8e234b16815e73a471779267: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:07.527653 12710 ts_tablet_manager.cc:1905] T 6280122ba0e34c41a231da403f70f861 P 0419aec032574ff49eee704aabbf60be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:07.528228 12711 ts_tablet_manager.cc:1905] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:07.534826 12409 tablet_service.cc:1467] Processing CreateTablet for tablet b254b717b1dc4821acb411571d0e27b8 (DEFAULT_TABLE table=test-table [id=b16cecf42e9648f38cd5a5c6c87dacb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:07.534823 12553 tablet_service.cc:1467] Processing CreateTablet for tablet b254b717b1dc4821acb411571d0e27b8 (DEFAULT_TABLE table=test-table [id=b16cecf42e9648f38cd5a5c6c87dacb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:07.535254 12709 ts_tablet_manager.cc:1918] T 6280122ba0e34c41a231da403f70f861 P ded8f46c8e234b16815e73a471779267: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:13:07.535331 12409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b254b717b1dc4821acb411571d0e27b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:07.535348 12553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b254b717b1dc4821acb411571d0e27b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:07.535485 12709 log.cc:1196] T 6280122ba0e34c41a231da403f70f861 P ded8f46c8e234b16815e73a471779267: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/6280122ba0e34c41a231da403f70f861
I20260927 15:13:07.535928 12710 ts_tablet_manager.cc:1918] T 6280122ba0e34c41a231da403f70f861 P 0419aec032574ff49eee704aabbf60be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:13:07.535980 12709 ts_tablet_manager.cc:1939] T 6280122ba0e34c41a231da403f70f861 P ded8f46c8e234b16815e73a471779267: Deleting consensus metadata
I20260927 15:13:07.537163 12710 log.cc:1196] T 6280122ba0e34c41a231da403f70f861 P 0419aec032574ff49eee704aabbf60be: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/6280122ba0e34c41a231da403f70f861
I20260927 15:13:07.537722 12710 ts_tablet_manager.cc:1939] T 6280122ba0e34c41a231da403f70f861 P 0419aec032574ff49eee704aabbf60be: Deleting consensus metadata
I20260927 15:13:07.538235 12118 catalog_manager.cc:4841] TS ded8f46c8e234b16815e73a471779267 (127.10.33.67:38171): tablet 6280122ba0e34c41a231da403f70f861 (table test-table [id=8a8169d57cab4eeda38336ecd086bb2e]) successfully deleted
I20260927 15:13:07.538308 12269 tablet_service.cc:1467] Processing CreateTablet for tablet b254b717b1dc4821acb411571d0e27b8 (DEFAULT_TABLE table=test-table [id=b16cecf42e9648f38cd5a5c6c87dacb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:07.539212 12269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b254b717b1dc4821acb411571d0e27b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:07.541636 12711 ts_tablet_manager.cc:1918] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:13:07.541824 12711 log.cc:1196] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/6280122ba0e34c41a231da403f70f861
I20260927 15:13:07.542454 12720 tablet_bootstrap.cc:492] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267: Bootstrap starting.
I20260927 15:13:07.542654 12711 ts_tablet_manager.cc:1939] T 6280122ba0e34c41a231da403f70f861 P 5049bfa6fecf4556873b696e965b888a: Deleting consensus metadata
I20260927 15:13:07.543928 12119 catalog_manager.cc:4841] TS 5049bfa6fecf4556873b696e965b888a (127.10.33.65:37067): tablet 6280122ba0e34c41a231da403f70f861 (table test-table [id=8a8169d57cab4eeda38336ecd086bb2e]) successfully deleted
I20260927 15:13:07.544121 12720 tablet_bootstrap.cc:654] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:07.544863 12120 catalog_manager.cc:4841] TS 0419aec032574ff49eee704aabbf60be (127.10.33.66:44521): tablet 6280122ba0e34c41a231da403f70f861 (table test-table [id=8a8169d57cab4eeda38336ecd086bb2e]) successfully deleted
I20260927 15:13:07.544922 12723 tablet_bootstrap.cc:492] T b254b717b1dc4821acb411571d0e27b8 P 0419aec032574ff49eee704aabbf60be: Bootstrap starting.
I20260927 15:13:07.546952 12723 tablet_bootstrap.cc:654] T b254b717b1dc4821acb411571d0e27b8 P 0419aec032574ff49eee704aabbf60be: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:07.548178 12722 tablet_bootstrap.cc:492] T b254b717b1dc4821acb411571d0e27b8 P 5049bfa6fecf4556873b696e965b888a: Bootstrap starting.
I20260927 15:13:07.550310 12722 tablet_bootstrap.cc:654] T b254b717b1dc4821acb411571d0e27b8 P 5049bfa6fecf4556873b696e965b888a: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:07.550524 12723 tablet_bootstrap.cc:492] T b254b717b1dc4821acb411571d0e27b8 P 0419aec032574ff49eee704aabbf60be: No bootstrap required, opened a new log
I20260927 15:13:07.550736 12723 ts_tablet_manager.cc:1397] T b254b717b1dc4821acb411571d0e27b8 P 0419aec032574ff49eee704aabbf60be: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260927 15:13:07.551275 12720 tablet_bootstrap.cc:492] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267: No bootstrap required, opened a new log
I20260927 15:13:07.551467 12720 ts_tablet_manager.cc:1397] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260927 15:13:07.551633 12723 raft_consensus.cc:348] T b254b717b1dc4821acb411571d0e27b8 P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:07.552006 12723 raft_consensus.cc:374] T b254b717b1dc4821acb411571d0e27b8 P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:07.552103 12723 raft_consensus.cc:729] T b254b717b1dc4821acb411571d0e27b8 P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0419aec032574ff49eee704aabbf60be, State: Initialized, Role: FOLLOWER
I20260927 15:13:07.552299 12720 raft_consensus.cc:348] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:07.552384 12723 consensus_queue.cc:260] T b254b717b1dc4821acb411571d0e27b8 P 0419aec032574ff49eee704aabbf60be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:07.552513 12720 raft_consensus.cc:374] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:07.552592 12720 raft_consensus.cc:729] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded8f46c8e234b16815e73a471779267, State: Initialized, Role: FOLLOWER
I20260927 15:13:07.552882 12720 consensus_queue.cc:260] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:07.553072 12722 tablet_bootstrap.cc:492] T b254b717b1dc4821acb411571d0e27b8 P 5049bfa6fecf4556873b696e965b888a: No bootstrap required, opened a new log
I20260927 15:13:07.553265 12722 ts_tablet_manager.cc:1397] T b254b717b1dc4821acb411571d0e27b8 P 5049bfa6fecf4556873b696e965b888a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:13:07.553678 12723 ts_tablet_manager.cc:1428] T b254b717b1dc4821acb411571d0e27b8 P 0419aec032574ff49eee704aabbf60be: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260927 15:13:07.554162 12722 raft_consensus.cc:348] T b254b717b1dc4821acb411571d0e27b8 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:07.554332 12720 ts_tablet_manager.cc:1428] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 15:13:07.554419 12722 raft_consensus.cc:374] T b254b717b1dc4821acb411571d0e27b8 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:07.554507 12722 raft_consensus.cc:729] T b254b717b1dc4821acb411571d0e27b8 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5049bfa6fecf4556873b696e965b888a, State: Initialized, Role: FOLLOWER
I20260927 15:13:07.554749 12722 consensus_queue.cc:260] T b254b717b1dc4821acb411571d0e27b8 P 5049bfa6fecf4556873b696e965b888a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:07.555282 12722 ts_tablet_manager.cc:1428] T b254b717b1dc4821acb411571d0e27b8 P 5049bfa6fecf4556873b696e965b888a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:13:07.646732 12724 raft_consensus.cc:482] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:13:07.647018 12724 raft_consensus.cc:504] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:07.648891 12724 leader_election.cc:288] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5049bfa6fecf4556873b696e965b888a (127.10.33.65:37067), 0419aec032574ff49eee704aabbf60be (127.10.33.66:44521)
I20260927 15:13:07.657642 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:07.649413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:38149 (local address 127.10.33.65:37067)
0927 15:13:07.649839 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:07.649848 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:07.650943 (+ 1095us) server_negotiation.cc:407] Connection header received
0927 15:13:07.651271 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:07.651277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:07.651374 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:07.651524 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:07.652222 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:07.653417 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:07.655793 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:07.656633 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:07.656812 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:07.656898 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:07.657227 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 15:13:07.657366 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260927 15:13:07.657848 12738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:07.649230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:37067 (local address 127.10.33.67:38149)
0927 15:13:07.650741 (+ 1511us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:07.650768 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:07.651025 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:07.651586 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:07.651592 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:07.651620 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:07.652022 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:07.652037 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:07.653616 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:07.653621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:07.655639 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:07.655652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:07.656825 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:07.656832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:07.657068 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:07.657097 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 15:13:07.657183 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 15:13:07.657341 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1348,"mutex_wait_us":28,"thread_start_us":1135,"threads_started":1}
I20260927 15:13:07.657845 12739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:07.649458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:34567 (local address 127.10.33.66:44521)
0927 15:13:07.650439 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:07.650446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:07.651547 (+ 1101us) server_negotiation.cc:407] Connection header received
0927 15:13:07.651749 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:07.651754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:07.651866 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:07.651999 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:07.652806 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:07.653708 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:07.655928 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:07.656746 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:07.656905 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:07.656981 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:07.657347 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 15:13:07.657476 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":596,"threads_started":1}
I20260927 15:13:07.658377 12740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:07.649465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:44521 (local address 127.10.33.67:34567)
0927 15:13:07.651409 (+ 1944us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:07.651430 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:07.651572 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:07.652104 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:07.652108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:07.652141 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:07.652618 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:07.652632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:07.653904 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:07.653909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:07.655777 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:07.655790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:07.656969 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:07.656975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:07.657190 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:07.657219 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 15:13:07.657304 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 15:13:07.657460 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1757,"mutex_wait_us":47,"thread_start_us":1541,"threads_started":1}
I20260927 15:13:07.658937 12290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b254b717b1dc4821acb411571d0e27b8" candidate_uuid: "ded8f46c8e234b16815e73a471779267" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5049bfa6fecf4556873b696e965b888a" is_pre_election: true
I20260927 15:13:07.659392 12290 raft_consensus.cc:2454] T b254b717b1dc4821acb411571d0e27b8 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded8f46c8e234b16815e73a471779267 in term 0.
I20260927 15:13:07.660089 12501 leader_election.cc:302] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5049bfa6fecf4556873b696e965b888a, ded8f46c8e234b16815e73a471779267; no voters:
I20260927 15:13:07.660542 12724 raft_consensus.cc:2781] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:13:07.660672 12724 raft_consensus.cc:482] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:13:07.660637 12428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b254b717b1dc4821acb411571d0e27b8" candidate_uuid: "ded8f46c8e234b16815e73a471779267" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0419aec032574ff49eee704aabbf60be" is_pre_election: true
I20260927 15:13:07.660796 12724 raft_consensus.cc:3037] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:07.660905 12428 raft_consensus.cc:2454] T b254b717b1dc4821acb411571d0e27b8 P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded8f46c8e234b16815e73a471779267 in term 0.
I20260927 15:13:07.663295 12724 raft_consensus.cc:504] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:07.663985 12724 leader_election.cc:288] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267 [CANDIDATE]: Term 1 election: Requested vote from peers 5049bfa6fecf4556873b696e965b888a (127.10.33.65:37067), 0419aec032574ff49eee704aabbf60be (127.10.33.66:44521)
I20260927 15:13:07.664599 12428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b254b717b1dc4821acb411571d0e27b8" candidate_uuid: "ded8f46c8e234b16815e73a471779267" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0419aec032574ff49eee704aabbf60be"
I20260927 15:13:07.664603 12290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b254b717b1dc4821acb411571d0e27b8" candidate_uuid: "ded8f46c8e234b16815e73a471779267" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5049bfa6fecf4556873b696e965b888a"
I20260927 15:13:07.664803 12428 raft_consensus.cc:3037] T b254b717b1dc4821acb411571d0e27b8 P 0419aec032574ff49eee704aabbf60be [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:07.664830 12290 raft_consensus.cc:3037] T b254b717b1dc4821acb411571d0e27b8 P 5049bfa6fecf4556873b696e965b888a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:07.667008 12428 raft_consensus.cc:2454] T b254b717b1dc4821acb411571d0e27b8 P 0419aec032574ff49eee704aabbf60be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded8f46c8e234b16815e73a471779267 in term 1.
I20260927 15:13:07.667192 12290 raft_consensus.cc:2454] T b254b717b1dc4821acb411571d0e27b8 P 5049bfa6fecf4556873b696e965b888a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded8f46c8e234b16815e73a471779267 in term 1.
I20260927 15:13:07.667532 12502 leader_election.cc:302] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0419aec032574ff49eee704aabbf60be, ded8f46c8e234b16815e73a471779267; no voters:
I20260927 15:13:07.667891 12724 raft_consensus.cc:2781] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:07.668372 12724 raft_consensus.cc:686] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267 [term 1 LEADER]: Becoming Leader. State: Replica: ded8f46c8e234b16815e73a471779267, State: Running, Role: LEADER
I20260927 15:13:07.668833 12724 consensus_queue.cc:237] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } }
I20260927 15:13:07.672773 12132 catalog_manager.cc:5495] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267 reported cstate change: term changed from 0 to 1, leader changed from <none> to ded8f46c8e234b16815e73a471779267 (127.10.33.67). New cstate: current_term: 1 leader_uuid: "ded8f46c8e234b16815e73a471779267" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded8f46c8e234b16815e73a471779267" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 38171 } health_report { overall_health: HEALTHY } } }
I20260927 15:13:07.700428 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:07.693258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.10.33.67:38171)
0927 15:13:07.693994 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:07.693998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:07.694018 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:13:07.694116 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:07.694120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:07.694261 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:07.694434 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:13:07.694913 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:07.695736 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:07.697116 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:07.697573 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:07.697733 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:07.698493 (+ 760us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:13:07.699397 (+ 904us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:13:07.699541 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:07.699860 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 15:13:07.699982 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":417,"threads_started":1}
I20260927 15:13:07.719787 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:07.714036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:53553 (local address 127.10.33.65:37067)
0927 15:13:07.714422 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:07.714430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:07.714929 (+ 499us) server_negotiation.cc:407] Connection header received
0927 15:13:07.715143 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:07.715148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:07.715236 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:07.715317 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:13:07.715816 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:07.716635 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:07.718046 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:07.718475 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:07.718589 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:07.719026 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:13:07.719150 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:13:07.719224 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:07.719358 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 15:13:07.719444 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 15:13:07.719849 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:07.713948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:37067 (local address 127.10.33.67:53553)
0927 15:13:07.714796 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:07.714820 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:07.714997 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:07.715344 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:07.715349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:07.715360 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:13:07.715683 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:07.715692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:07.716773 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:07.716779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:07.717926 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:07.717939 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:07.718654 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:07.718661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:07.718790 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:07.718852 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:13:07.719241 (+ 389us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:13:07.719253 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 15:13:07.719322 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 15:13:07.719474 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":458,"threads_started":1}
I20260927 15:13:07.729995 12428 raft_consensus.cc:1261] T b254b717b1dc4821acb411571d0e27b8 P 0419aec032574ff49eee704aabbf60be [term 1 FOLLOWER]: Refusing update from remote peer ded8f46c8e234b16815e73a471779267: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:07.730453 12290 raft_consensus.cc:1261] T b254b717b1dc4821acb411571d0e27b8 P 5049bfa6fecf4556873b696e965b888a [term 1 FOLLOWER]: Refusing update from remote peer ded8f46c8e234b16815e73a471779267: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:07.730737 12741 consensus_queue.cc:1035] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0419aec032574ff49eee704aabbf60be" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:07.731169 12724 consensus_queue.cc:1035] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5049bfa6fecf4556873b696e965b888a" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:07.735762 12693 mvcc.cc:205] Tried to move back new op lower bound from 7333978061735788544 to 7333978061495836672. Current Snapshot: MvccSnapshot[applied={T|T < 7333978061735788544}]
I20260927 15:13:07.761004 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:07.754696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:32771 (local address 127.10.33.67:38171)
0927 15:13:07.755051 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:07.755056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:07.755581 (+ 525us) server_negotiation.cc:407] Connection header received
0927 15:13:07.755784 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:07.755790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:07.755890 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:07.756008 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:13:07.756609 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:07.757422 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:07.759090 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:07.759654 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:07.759799 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:07.760298 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:13:07.760432 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:13:07.760523 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:07.760644 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 15:13:07.760722 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260927 15:13:07.761046 12750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:07.754650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:38171 (local address 127.10.33.65:32771)
0927 15:13:07.755463 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:07.755481 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:07.755622 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:07.756038 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:07.756042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:07.756055 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:13:07.756457 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:07.756468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:07.757565 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:07.757569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:07.758957 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:07.758973 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:07.759860 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:07.759868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:07.760053 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:07.760107 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:13:07.760537 (+ 430us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:13:07.760546 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 15:13:07.760616 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 15:13:07.760745 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":436,"threads_started":1}
W20260927 15:13:07.762323 12693 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 15:13:09.367874 12132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:09.368104 12132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:09.372159 12132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f21f133ccd8c4bca9ec1645ef4be01d0: Sending DeleteTablet for 3 replicas of tablet b254b717b1dc4821acb411571d0e27b8
I20260927 15:13:09.372951 12409 tablet_service.cc:1514] Processing DeleteTablet for tablet b254b717b1dc4821acb411571d0e27b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:09 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:59360
I20260927 15:13:09.372968 12269 tablet_service.cc:1514] Processing DeleteTablet for tablet b254b717b1dc4821acb411571d0e27b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:09 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:46190
I20260927 15:13:09.373055 12553 tablet_service.cc:1514] Processing DeleteTablet for tablet b254b717b1dc4821acb411571d0e27b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:09 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:38224
I20260927 15:13:09.373805 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 12204
I20260927 15:13:09.376348 12777 tablet_replica.cc:330] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267: stopping tablet replica
I20260927 15:13:09.376535 12776 tablet_replica.cc:330] T b254b717b1dc4821acb411571d0e27b8 P 0419aec032574ff49eee704aabbf60be: stopping tablet replica
I20260927 15:13:09.376617 12777 raft_consensus.cc:2229] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:13:09.376714 12776 raft_consensus.cc:2229] T b254b717b1dc4821acb411571d0e27b8 P 0419aec032574ff49eee704aabbf60be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:09.376937 12776 raft_consensus.cc:2258] T b254b717b1dc4821acb411571d0e27b8 P 0419aec032574ff49eee704aabbf60be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:09.377182 12777 raft_consensus.cc:2258] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:09.378433 12776 ts_tablet_manager.cc:1905] T b254b717b1dc4821acb411571d0e27b8 P 0419aec032574ff49eee704aabbf60be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:09.378613 12777 ts_tablet_manager.cc:1905] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:09.382601 12775 tablet_replica.cc:330] T b254b717b1dc4821acb411571d0e27b8 P 5049bfa6fecf4556873b696e965b888a: stopping tablet replica
I20260927 15:13:09.382822 12775 raft_consensus.cc:2229] T b254b717b1dc4821acb411571d0e27b8 P 5049bfa6fecf4556873b696e965b888a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:09.383026 12775 raft_consensus.cc:2258] T b254b717b1dc4821acb411571d0e27b8 P 5049bfa6fecf4556873b696e965b888a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:09.384128 12775 ts_tablet_manager.cc:1905] T b254b717b1dc4821acb411571d0e27b8 P 5049bfa6fecf4556873b696e965b888a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:09.385208 12776 ts_tablet_manager.cc:1918] T b254b717b1dc4821acb411571d0e27b8 P 0419aec032574ff49eee704aabbf60be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:13:09.385346 12776 log.cc:1196] T b254b717b1dc4821acb411571d0e27b8 P 0419aec032574ff49eee704aabbf60be: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/b254b717b1dc4821acb411571d0e27b8
I20260927 15:13:09.385751 12776 ts_tablet_manager.cc:1939] T b254b717b1dc4821acb411571d0e27b8 P 0419aec032574ff49eee704aabbf60be: Deleting consensus metadata
I20260927 15:13:09.386572 12777 ts_tablet_manager.cc:1918] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:13:09.386701 12777 log.cc:1196] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/b254b717b1dc4821acb411571d0e27b8
I20260927 15:13:09.386911 12120 catalog_manager.cc:4841] TS 0419aec032574ff49eee704aabbf60be (127.10.33.66:44521): tablet b254b717b1dc4821acb411571d0e27b8 (table test-table [id=b16cecf42e9648f38cd5a5c6c87dacb9]) successfully deleted
I20260927 15:13:09.387109 12777 ts_tablet_manager.cc:1939] T b254b717b1dc4821acb411571d0e27b8 P ded8f46c8e234b16815e73a471779267: Deleting consensus metadata
I20260927 15:13:09.387991 12118 catalog_manager.cc:4841] TS ded8f46c8e234b16815e73a471779267 (127.10.33.67:38171): tablet b254b717b1dc4821acb411571d0e27b8 (table test-table [id=b16cecf42e9648f38cd5a5c6c87dacb9]) successfully deleted
I20260927 15:13:09.390987 12775 ts_tablet_manager.cc:1918] T b254b717b1dc4821acb411571d0e27b8 P 5049bfa6fecf4556873b696e965b888a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:13:09.391129 12775 log.cc:1196] T b254b717b1dc4821acb411571d0e27b8 P 5049bfa6fecf4556873b696e965b888a: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/b254b717b1dc4821acb411571d0e27b8
I20260927 15:13:09.391571 12775 ts_tablet_manager.cc:1939] T b254b717b1dc4821acb411571d0e27b8 P 5049bfa6fecf4556873b696e965b888a: Deleting consensus metadata
I20260927 15:13:09.392701 12119 catalog_manager.cc:4841] TS 5049bfa6fecf4556873b696e965b888a (127.10.33.65:37067): tablet b254b717b1dc4821acb411571d0e27b8 (table test-table [id=b16cecf42e9648f38cd5a5c6c87dacb9]) successfully deleted
Sep 27 15:13:09 dist-test-slave-f5h6 krb5kdc[12088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521987, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.33.65@KRBTEST.COM
I20260927 15:13:09.399838 12779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:09.376056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46578 (local address 127.10.33.65:37067)
0927 15:13:09.382777 (+ 6721us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:09.382785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:09.383546 (+ 761us) server_negotiation.cc:407] Connection header received
0927 15:13:09.383609 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:09.383614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:09.383696 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:09.383791 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:09.385659 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:09.386873 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:09.388563 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:09.389065 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:09.389176 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:09.394029 (+ 4853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:09.394044 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:09.394062 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:09.394105 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:09.396933 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:09.397664 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:09.397675 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:09.397678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:09.397830 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:09.398301 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:09.398306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:09.398310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:09.398811 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:09.399044 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:09.399341 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 15:13:09.399584 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6582,"thread_start_us":6429,"threads_started":1}
W20260927 15:13:09.400494 12335 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46578
W20260927 15:13:09.401059 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 12204: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:13:09.401259 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 12204
I20260927 15:13:09.421972 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 12344
Sep 27 15:13:09 dist-test-slave-f5h6 krb5kdc[12088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521987, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.33.66@KRBTEST.COM
I20260927 15:13:09.437218 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:09.422677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45426 (local address 127.10.33.66:44521)
0927 15:13:09.423180 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:09.423184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:09.423199 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:13:09.423251 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:09.423255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:09.423320 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:09.423421 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:09.424639 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:09.425386 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:09.426750 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:09.427154 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:09.427273 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:09.431510 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:09.431528 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:09.431543 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:09.431582 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:09.434325 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:09.435059 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:09.435067 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:09.435069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:09.435231 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:09.435683 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:09.435687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:09.435689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:09.436077 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:09.436400 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:09.436669 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 15:13:09.436857 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":228,"threads_started":1}
W20260927 15:13:09.437975 12474 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45426
W20260927 15:13:09.438623 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 12344: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:13:09.438773 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 12344
I20260927 15:13:09.452517 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 12483
Sep 27 15:13:09 dist-test-slave-f5h6 krb5kdc[12088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521987, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.33.67@KRBTEST.COM
I20260927 15:13:09.467684 12781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:09.453301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.10.33.67:38171)
0927 15:13:09.453763 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:09.453768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:09.453784 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:13:09.453837 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:09.453841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:09.453940 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:09.454034 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:09.455415 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:09.456196 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:09.457372 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:09.457770 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:09.457898 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:09.462047 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:09.462064 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:09.462080 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:09.462123 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:09.464929 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:09.465642 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:09.465649 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:09.465651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:09.465771 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:09.466269 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:09.466273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:09.466275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:09.466696 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:09.466922 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:09.467239 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 15:13:09.467409 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":231,"threads_started":1}
W20260927 15:13:09.468333 12613 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55760
W20260927 15:13:09.468912 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 12483: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:13:09.469090 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 12483
I20260927 15:13:09.486712 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 12101
I20260927 15:13:09.487950 12173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:09.624747 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 12101
2026-09-27T15:13:09Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8309 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:13:09 dist-test-slave-f5h6 krb5kdc[12785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:13:09 dist-test-slave-f5h6 krb5kdc[12785](info): set up 2 sockets
Sep 27 15:13:09 dist-test-slave-f5h6 krb5kdc[12785](info): commencing operation
krb5kdc: starting...
W20260927 15:13:10.646385 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 15:13:10.801725 10373 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.155s user 0.000s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260927 15:13:10.910884 10373 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.109s user 0.004s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260927 15:13:11.048792 10373 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.138s user 0.000s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/test-user.keytab.
W20260927 15:13:11.729707 10373 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.681s user 0.000s sys 0.004s
Sep 27 15:13:11 dist-test-slave-f5h6 krb5kdc[12785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:13:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:13:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
W20260927 15:13:13.053714 10373 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.10.33.126: real 1.275s user 0.000s sys 0.009s
I20260927 15:13:13.054731 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:33763
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:33763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:13.400010 12798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:13.400333 12798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:13.400446 12798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:13.408205 12798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:13:13.408330 12798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:13.408391 12798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:13.408466 12798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:13:13.408520 12798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:13:13.419126 12798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:33763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:33763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:13.420535 12798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:13.422021 12798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:13.434123 12803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:13.434296 12806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:13.434278 12804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:13.435086 12798 server_base.cc:1034] running on GCE node
I20260927 15:13:13.436076 12798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:13.437762 12798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:13.439002 12798 hybrid_clock.cc:648] HybridClock initialized: now 1790521993438944 us; error 53 us; skew 500 ppm
I20260927 15:13:13.439400 12798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:13 dist-test-slave-f5h6 krb5kdc[12785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521993, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:13.442929 12798 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:13:13.444479 12798 webserver.cc:460] Webserver started at http://127.10.33.126:41363/ using document root <none> and password file <none>
I20260927 15:13:13.445245 12798 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:13.445363 12798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:13.445657 12798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:13.448755 12798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "6b904c71cca649478957eab356002af2"
format_stamp: "Formatted at 2026-09-27 15:13:13 on dist-test-slave-f5h6"
server_key: "bd7348b5a8ba0ec9041af0ce59594b7d"
server_key_iv: "9c35336d32ce65cdf6111b911f7050ea"
server_key_version: "encryptionkey@0"
I20260927 15:13:13.449510 12798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "6b904c71cca649478957eab356002af2"
format_stamp: "Formatted at 2026-09-27 15:13:13 on dist-test-slave-f5h6"
server_key: "bd7348b5a8ba0ec9041af0ce59594b7d"
server_key_iv: "9c35336d32ce65cdf6111b911f7050ea"
server_key_version: "encryptionkey@0"
I20260927 15:13:13.454882 12798 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 15:13:13.458648 12813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:13.460188 12798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:13:13.460371 12798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "6b904c71cca649478957eab356002af2"
format_stamp: "Formatted at 2026-09-27 15:13:13 on dist-test-slave-f5h6"
server_key: "bd7348b5a8ba0ec9041af0ce59594b7d"
server_key_iv: "9c35336d32ce65cdf6111b911f7050ea"
server_key_version: "encryptionkey@0"
I20260927 15:13:13.460577 12798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:13.477998 12798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:13.482012 12798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:13.482283 12798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:13.501286 12798 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:33763
I20260927 15:13:13.501305 12864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:33763 every 8 connection(s)
I20260927 15:13:13.502700 12798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:13:13.503780 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 12798
I20260927 15:13:13.503984 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:13:13.504850 10373 external_mini_cluster.cc:1428] Setting key 9759629f829024e32e30dae473736157
I20260927 15:13:13.507738 12865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:13.518231 12865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2: Bootstrap starting.
Sep 27 15:13:13 dist-test-slave-f5h6 krb5kdc[12785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:13.521605 12865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:13.522735 12865 log.cc:824] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:13.525498 12865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2: No bootstrap required, opened a new log
I20260927 15:13:13.527014 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:13.507150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40512 (local address 127.10.33.126:33763)
0927 15:13:13.508032 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:13.508048 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:13.508104 (+ 56us) server_negotiation.cc:407] Connection header received
0927 15:13:13.509837 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:13.509898 (+ 61us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:13.510333 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:13.510720 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:13.512130 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:13.513484 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:13.514948 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:13.515687 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:13.515864 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:13.519966 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:13.519974 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:13.519989 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:13.520031 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:13.523132 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:13.523841 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:13.523848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:13.523855 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:13.524013 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:13.524401 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:13.524404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:13.524407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:13.524904 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:13.525181 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:13.525602 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 15:13:13.526040 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":360,"threads_started":1}
I20260927 15:13:13.533445 12865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b904c71cca649478957eab356002af2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 33763 } }
I20260927 15:13:13.533825 12865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:13.533941 12865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b904c71cca649478957eab356002af2, State: Initialized, Role: FOLLOWER
I20260927 15:13:13.534490 12865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b904c71cca649478957eab356002af2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 33763 } }
I20260927 15:13:13.534713 12865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:13:13.534832 12865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:13:13.534991 12865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:13.554126 12865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b904c71cca649478957eab356002af2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 33763 } }
I20260927 15:13:13.554930 12865 leader_election.cc:302] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b904c71cca649478957eab356002af2; no voters:
I20260927 15:13:13.555711 12865 leader_election.cc:288] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:13:13.555850 12870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:13.557466 12870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2 [term 1 LEADER]: Becoming Leader. State: Replica: 6b904c71cca649478957eab356002af2, State: Running, Role: LEADER
I20260927 15:13:13.558156 12870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b904c71cca649478957eab356002af2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 33763 } }
I20260927 15:13:13.559041 12865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:13:13.561022 12872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b904c71cca649478957eab356002af2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b904c71cca649478957eab356002af2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 33763 } } }
I20260927 15:13:13.560999 12871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b904c71cca649478957eab356002af2. Latest consensus state: current_term: 1 leader_uuid: "6b904c71cca649478957eab356002af2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b904c71cca649478957eab356002af2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 33763 } } }
I20260927 15:13:13.561461 12871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:13.561461 12872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:13.563153 12877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:13:13.569152 12877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:13:13.578325 12877 catalog_manager.cc:1347] Generated new cluster ID: 0b61d822aacf4ad09300283148347dff
I20260927 15:13:13.578508 12877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:13:13.595383 12877 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:13:13.597326 12877 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:13:13.609714 12877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b904c71cca649478957eab356002af2: Generated new TSK 0
I20260927 15:13:13.610958 12877 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.33.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
W20260927 15:13:15.002813 10373 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.10.33.65: real 1.377s user 0.003s sys 0.008s
I20260927 15:13:15.003859 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:33763
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:15.340719 12891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:15.341039 12891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:15.341168 12891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:15.349314 12891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:15.349464 12891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:15.349628 12891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:13:15.359843 12891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:33763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:15.361402 12891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:15.362896 12891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:15.376189 12899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:15.376174 12896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:15.376628 12897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:15.377185 12891 server_base.cc:1034] running on GCE node
I20260927 15:13:15.377925 12891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:15.378955 12891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:15.380173 12891 hybrid_clock.cc:648] HybridClock initialized: now 1790521995380123 us; error 43 us; skew 500 ppm
I20260927 15:13:15.380481 12891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:15 dist-test-slave-f5h6 krb5kdc[12785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521995, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:15.399561 12891 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:13:15.401232 12891 webserver.cc:460] Webserver started at http://127.10.33.65:38045/ using document root <none> and password file <none>
I20260927 15:13:15.402016 12891 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:15.402143 12891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:15.402462 12891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:15.405593 12891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "5c9e29400258417c901a099f41b4dec7"
format_stamp: "Formatted at 2026-09-27 15:13:15 on dist-test-slave-f5h6"
server_key: "1b1099165a43ba48942ae180dd17ae18"
server_key_iv: "316a4421b9353fca06502442b250b9b0"
server_key_version: "encryptionkey@0"
I20260927 15:13:15.406502 12891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "5c9e29400258417c901a099f41b4dec7"
format_stamp: "Formatted at 2026-09-27 15:13:15 on dist-test-slave-f5h6"
server_key: "1b1099165a43ba48942ae180dd17ae18"
server_key_iv: "316a4421b9353fca06502442b250b9b0"
server_key_version: "encryptionkey@0"
I20260927 15:13:15.412415 12891 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:13:15.416275 12906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:15.417932 12891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:13:15.418128 12891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "5c9e29400258417c901a099f41b4dec7"
format_stamp: "Formatted at 2026-09-27 15:13:15 on dist-test-slave-f5h6"
server_key: "1b1099165a43ba48942ae180dd17ae18"
server_key_iv: "316a4421b9353fca06502442b250b9b0"
server_key_version: "encryptionkey@0"
I20260927 15:13:15.418334 12891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:15.450662 12891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:15.455456 12891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:15.455763 12891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:15.456831 12891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:15.458276 12891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:15.458356 12891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:15.458470 12891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:15.458527 12891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:15.492031 12891 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:34655
I20260927 15:13:15.492043 13018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:34655 every 8 connection(s)
I20260927 15:13:15.493409 12891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:13:15.503583 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 12891
I20260927 15:13:15.503801 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:13:15.504531 10373 external_mini_cluster.cc:1428] Setting key 313ab33c70699062be00cbaaf73d8432
Sep 27 15:13:15 dist-test-slave-f5h6 krb5kdc[12785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521995, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:15.514590 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:15.496318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:40939 (local address 127.10.33.126:33763)
0927 15:13:15.496824 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:15.496829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:15.497761 (+ 932us) server_negotiation.cc:407] Connection header received
0927 15:13:15.499751 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:15.499757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:15.499845 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:15.499958 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:15.502278 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:15.503075 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:15.504839 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:15.505355 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:15.505467 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:15.509075 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:15.509080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:15.509084 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:15.509125 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:15.511545 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:15.512299 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:15.512303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:15.512306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:15.512425 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:15.512851 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:15.512855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:15.512857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:15.513140 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:15.513284 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:15.514000 (+ 716us) server_negotiation.cc:299] Negotiation successful
0927 15:13:15.514235 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":229,"threads_started":1}
I20260927 15:13:15.515553 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:15.496873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:33763 (local address 127.10.33.65:40939)
0927 15:13:15.497589 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:15.497633 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:15.499363 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:15.500315 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:15.500331 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:15.501166 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:15.502024 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:15.502051 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:15.503266 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:15.503270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:15.504693 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:15.504706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:15.505582 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:15.505590 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:15.505846 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:15.506589 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:15.506671 (+ 82us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:15.508831 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:15.511719 (+ 2888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:15.511728 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:15.511743 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:15.512171 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:15.512529 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:15.512534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:15.512538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:15.512735 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:15.513268 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:15.513277 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:15.513630 (+ 353us) client_negotiation.cc:769] Sending connection context
0927 15:13:15.513899 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 15:13:15.514434 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":296,"threads_started":1}
I20260927 15:13:15.517347 13019 heartbeater.cc:347] Connected to a master server at 127.10.33.126:33763
I20260927 15:13:15.517711 13019 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:15.518527 13019 heartbeater.cc:511] Master 127.10.33.126:33763 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.33.66@KRBTEST.COM; defaulting to no policy
I20260927 15:13:15.520759 12830 ts_manager.cc:194] Registered new tserver with Master: 5c9e29400258417c901a099f41b4dec7 (127.10.33.65:34655)
I20260927 15:13:15.522449 12830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:40939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
I20260927 15:13:15.546202 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:33763
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:15.865293 13025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:15.865587 13025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:15.865710 13025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:15.873078 13025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:15.873219 13025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:15.873332 13025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:13:15.883550 13025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:33763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:15.884832 13025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:15.886165 13025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:15.899647 13031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:15.899595 13033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:15.899621 13030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:15.900460 13025 server_base.cc:1034] running on GCE node
I20260927 15:13:15.901098 13025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:15.902130 13025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:15.903345 13025 hybrid_clock.cc:648] HybridClock initialized: now 1790521995903283 us; error 53 us; skew 500 ppm
I20260927 15:13:15.903653 13025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:15 dist-test-slave-f5h6 krb5kdc[12785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521995, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:15.906800 13025 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:13:15.908213 13025 webserver.cc:460] Webserver started at http://127.10.33.66:36569/ using document root <none> and password file <none>
I20260927 15:13:15.908865 13025 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:15.908962 13025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:15.909258 13025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:15.912331 13025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "85b367d1da574bd29db3b88382ecc9ca"
format_stamp: "Formatted at 2026-09-27 15:13:15 on dist-test-slave-f5h6"
server_key: "34ad5d053c17a5870209942213c658f0"
server_key_iv: "0eb39d0a189937ab11791b5bbf42ef14"
server_key_version: "encryptionkey@0"
I20260927 15:13:15.913151 13025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "85b367d1da574bd29db3b88382ecc9ca"
format_stamp: "Formatted at 2026-09-27 15:13:15 on dist-test-slave-f5h6"
server_key: "34ad5d053c17a5870209942213c658f0"
server_key_iv: "0eb39d0a189937ab11791b5bbf42ef14"
server_key_version: "encryptionkey@0"
I20260927 15:13:15.918725 13025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 15:13:15.922356 13040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:15.923815 13025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 15:13:15.923959 13025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "85b367d1da574bd29db3b88382ecc9ca"
format_stamp: "Formatted at 2026-09-27 15:13:15 on dist-test-slave-f5h6"
server_key: "34ad5d053c17a5870209942213c658f0"
server_key_iv: "0eb39d0a189937ab11791b5bbf42ef14"
server_key_version: "encryptionkey@0"
I20260927 15:13:15.924140 13025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:15.947952 13025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:15.952587 13025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:15.952862 13025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:15.953881 13025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:15.955224 13025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:15.955295 13025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:15.955379 13025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:15.955420 13025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:15.987183 13025 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:39511
I20260927 15:13:15.987226 13152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:39511 every 8 connection(s)
I20260927 15:13:15.988440 13025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:13:15.994735 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 13025
I20260927 15:13:15.994925 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:13:15.995452 10373 external_mini_cluster.cc:1428] Setting key 1e87772f163d8fad2823be0839ec72da
Sep 27 15:13:16 dist-test-slave-f5h6 krb5kdc[12785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521995, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:16.011494 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:15.991150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:59989 (local address 127.10.33.126:33763)
0927 15:13:15.991395 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:15.991402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:15.992705 (+ 1303us) server_negotiation.cc:407] Connection header received
0927 15:13:15.995148 (+ 2443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:15.995155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:15.995350 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:15.995559 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:15.997722 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:15.998501 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:16.000755 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:16.001171 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:16.001273 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:16.005233 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:16.005239 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:16.005244 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:16.005281 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:16.008314 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:16.009040 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:16.009045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:16.009048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:16.009189 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:16.009642 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:16.009647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:16.009651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:16.010044 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:16.010190 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:16.010920 (+ 730us) server_negotiation.cc:299] Negotiation successful
0927 15:13:16.011151 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:13:16.012434 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:15.991719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:33763 (local address 127.10.33.66:59989)
0927 15:13:15.992533 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:15.992574 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:15.994670 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:15.995983 (+ 1313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:15.995996 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:15.996730 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:15.997512 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:15.997541 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:15.998682 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:15.998688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:16.000580 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:16.000607 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:16.001402 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:16.001409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:16.001705 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:16.002415 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:16.002442 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:16.004986 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:16.008456 (+ 3470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:16.008463 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:16.008478 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:16.008929 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:16.009318 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:16.009322 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:16.009324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:16.009523 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:16.010173 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:16.010180 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:16.010534 (+ 354us) client_negotiation.cc:769] Sending connection context
0927 15:13:16.010805 (+ 271us) client_negotiation.cc:240] Negotiation successful
0927 15:13:16.011227 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":348,"threads_started":1}
WARNING: no policy specified for kudu/127.10.33.67@KRBTEST.COM; defaulting to no policy
I20260927 15:13:16.013914 13153 heartbeater.cc:347] Connected to a master server at 127.10.33.126:33763
I20260927 15:13:16.014253 13153 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:16.015030 13153 heartbeater.cc:511] Master 127.10.33.126:33763 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.67@KRBTEST.COM" created.
I20260927 15:13:16.016639 12830 ts_manager.cc:194] Registered new tserver with Master: 85b367d1da574bd29db3b88382ecc9ca (127.10.33.66:39511)
I20260927 15:13:16.017412 12830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:59989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
I20260927 15:13:16.037667 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:33763
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:16.356819 13158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:16.357136 13158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:16.357261 13158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:16.365013 13158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:16.365144 13158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:16.365293 13158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:13:16.375178 13158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:33763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:16.376469 13158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:16.377746 13158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:16.391021 13163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:16.391086 13164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:16.391058 13166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:16.391763 13158 server_base.cc:1034] running on GCE node
I20260927 15:13:16.392424 13158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:16.393419 13158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:16.394635 13158 hybrid_clock.cc:648] HybridClock initialized: now 1790521996394575 us; error 49 us; skew 500 ppm
I20260927 15:13:16.394958 13158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:16 dist-test-slave-f5h6 krb5kdc[12785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521996, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:16.398095 13158 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:13:16.399647 13158 webserver.cc:460] Webserver started at http://127.10.33.67:44055/ using document root <none> and password file <none>
I20260927 15:13:16.400341 13158 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:16.400455 13158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:16.400740 13158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:16.403764 13158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "6c50df8a74314e30b48880a1271845b9"
format_stamp: "Formatted at 2026-09-27 15:13:16 on dist-test-slave-f5h6"
server_key: "75c7e0b33366343b50bb98dcaa4785f3"
server_key_iv: "0fd0081b9c9331cb7426a4ae69a32fc7"
server_key_version: "encryptionkey@0"
I20260927 15:13:16.404579 13158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "6c50df8a74314e30b48880a1271845b9"
format_stamp: "Formatted at 2026-09-27 15:13:16 on dist-test-slave-f5h6"
server_key: "75c7e0b33366343b50bb98dcaa4785f3"
server_key_iv: "0fd0081b9c9331cb7426a4ae69a32fc7"
server_key_version: "encryptionkey@0"
I20260927 15:13:16.410210 13158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 15:13:16.413945 13173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:16.415534 13158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:13:16.415710 13158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "6c50df8a74314e30b48880a1271845b9"
format_stamp: "Formatted at 2026-09-27 15:13:16 on dist-test-slave-f5h6"
server_key: "75c7e0b33366343b50bb98dcaa4785f3"
server_key_iv: "0fd0081b9c9331cb7426a4ae69a32fc7"
server_key_version: "encryptionkey@0"
I20260927 15:13:16.415916 13158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:16.437991 13158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:16.442163 13158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:16.442430 13158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:16.443415 13158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:16.444732 13158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:16.444811 13158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:16.444922 13158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:16.444989 13158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:16.475455 13158 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:37331
I20260927 15:13:16.475497 13285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:37331 every 8 connection(s)
I20260927 15:13:16.476691 13158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:13:16.485282 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 13158
I20260927 15:13:16.485446 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:13:16.485848 10373 external_mini_cluster.cc:1428] Setting key 5fedca99194c1e117a91b2f6806dafd9
Sep 27 15:13:16 dist-test-slave-f5h6 krb5kdc[12785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521996, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:16.499794 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:16.479347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:51861 (local address 127.10.33.126:33763)
0927 15:13:16.479548 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:16.479554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:16.480751 (+ 1197us) server_negotiation.cc:407] Connection header received
0927 15:13:16.482936 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:16.482941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:16.483025 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:16.483132 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:16.484945 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:16.485826 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:16.488243 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:16.488935 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:16.489073 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:16.493185 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:16.493191 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:16.493194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:16.493229 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:16.496438 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:16.497396 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:16.497400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:16.497403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:16.497536 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:16.498002 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:16.498007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:16.498009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:16.498307 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:16.498450 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:16.499207 (+ 757us) server_negotiation.cc:299] Negotiation successful
0927 15:13:16.499442 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:13:16.501034 13288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:16.479790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:33763 (local address 127.10.33.67:51861)
0927 15:13:16.480554 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:16.480594 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:16.482522 (+ 1928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:16.483493 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:16.483503 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:16.484080 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:16.484726 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:16.484750 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:16.486047 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:16.486053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:16.488040 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:16.488060 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:16.489268 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:16.489279 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:16.489562 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:16.490592 (+ 1030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:16.490678 (+ 86us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:16.492908 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:16.496688 (+ 3780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:16.496697 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:16.496712 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:16.497259 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:16.497658 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:16.497663 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:16.497667 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:16.497890 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:16.498452 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:16.498461 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:16.498824 (+ 363us) client_negotiation.cc:769] Sending connection context
0927 15:13:16.499076 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 15:13:16.499654 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":320,"threads_started":1}
I20260927 15:13:16.502704 13286 heartbeater.cc:347] Connected to a master server at 127.10.33.126:33763
I20260927 15:13:16.502979 13286 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:16.503747 13286 heartbeater.cc:511] Master 127.10.33.126:33763 requested a full tablet report, sending...
I20260927 15:13:16.505216 12829 ts_manager.cc:194] Registered new tserver with Master: 6c50df8a74314e30b48880a1271845b9 (127.10.33.67:37331)
I20260927 15:13:16.506255 12829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:51861
I20260927 15:13:16.514557 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:13:16.526588 13019 heartbeater.cc:503] Master 127.10.33.126:33763 was elected leader, sending a full tablet report...
W20260927 15:13:16.542743 13021 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:13:16.540342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40520 (local address 127.10.33.126:33763)
0927 15:13:16.540580 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:16.540596 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:16.540892 (+ 296us) server_negotiation.cc:407] Connection header received
0927 15:13:16.541073 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:16.541077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:16.541148 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:16.541235 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:16.542535 (+ 1300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40520: BlockingRecv error: recv got EOF from 127.0.0.1:40520 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 15:13:16.545876 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 12891
W20260927 15:13:16.548602 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 12891: Not authorized: Client connection negotiation failed: client connection to 127.10.33.65:34655: server requires authentication, but client does not have Kerberos credentials available
I20260927 15:13:16.548806 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 12891
I20260927 15:13:16.560791 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 13025
W20260927 15:13:16.563454 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 13025: Not authorized: Client connection negotiation failed: client connection to 127.10.33.66:39511: server requires authentication, but client does not have Kerberos credentials available
I20260927 15:13:16.563637 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 13025
I20260927 15:13:16.575376 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 13158
W20260927 15:13:16.578018 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 13158: Not authorized: Client connection negotiation failed: client connection to 127.10.33.67:37331: server requires authentication, but client does not have Kerberos credentials available
I20260927 15:13:16.578157 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 13158
I20260927 15:13:16.590453 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 12798
I20260927 15:13:16.591750 12860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:16.696606 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 12798
2026-09-27T15:13:16Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (7069 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:13:16 dist-test-slave-f5h6 krb5kdc[13303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:13:16 dist-test-slave-f5h6 krb5kdc[13303](info): set up 2 sockets
Sep 27 15:13:16 dist-test-slave-f5h6 krb5kdc[13303](info): commencing operation
krb5kdc: starting...
W20260927 15:13:17.694249 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:13:17 dist-test-slave-f5h6 krb5kdc[13303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:13:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:13:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:13:17.882586 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:37243
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:35217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:37243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:18.206533 13316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:18.206836 13316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:18.206967 13316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:18.214178 13316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:13:18.214311 13316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:18.214366 13316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:18.214443 13316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:13:18.214504 13316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:13:18.224684 13316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:35217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:37243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:37243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:18.226085 13316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:18.227375 13316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:18.239054 13321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:18.239029 13322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:18.238986 13324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:18.240025 13316 server_base.cc:1034] running on GCE node
I20260927 15:13:18.240760 13316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:18.242216 13316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:18.243417 13316 hybrid_clock.cc:648] HybridClock initialized: now 1790521998243381 us; error 43 us; skew 500 ppm
I20260927 15:13:18.243731 13316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:18 dist-test-slave-f5h6 krb5kdc[13303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521998, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:18.246829 13316 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:13:18.248268 13316 webserver.cc:460] Webserver started at http://127.10.33.126:41197/ using document root <none> and password file <none>
I20260927 15:13:18.248935 13316 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:18.249049 13316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:18.249347 13316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:18.252318 13316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "c3177e49c4624f728737775fdae6688c"
format_stamp: "Formatted at 2026-09-27 15:13:18 on dist-test-slave-f5h6"
server_key: "670e4e6f6966c683d875de8435519624"
server_key_iv: "6a23a9278469004caf3af75b650e4147"
server_key_version: "encryptionkey@0"
I20260927 15:13:18.253104 13316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "c3177e49c4624f728737775fdae6688c"
format_stamp: "Formatted at 2026-09-27 15:13:18 on dist-test-slave-f5h6"
server_key: "670e4e6f6966c683d875de8435519624"
server_key_iv: "6a23a9278469004caf3af75b650e4147"
server_key_version: "encryptionkey@0"
I20260927 15:13:18.258548 13316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 15:13:18.262074 13331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:18.263442 13316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:13:18.263607 13316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "c3177e49c4624f728737775fdae6688c"
format_stamp: "Formatted at 2026-09-27 15:13:18 on dist-test-slave-f5h6"
server_key: "670e4e6f6966c683d875de8435519624"
server_key_iv: "6a23a9278469004caf3af75b650e4147"
server_key_version: "encryptionkey@0"
I20260927 15:13:18.263790 13316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:18.290740 13316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:18.295305 13316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:18.295580 13316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:18.313062 13316 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:37243
I20260927 15:13:18.313102 13382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:37243 every 8 connection(s)
I20260927 15:13:18.314495 13316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:13:18.318692 13383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:18.321614 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 13316
I20260927 15:13:18.321954 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:13:18.322726 10373 external_mini_cluster.cc:1428] Setting key 4d246445434ceca9f25ff4ae1f7bbc0e
I20260927 15:13:18.331710 13383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c: Bootstrap starting.
I20260927 15:13:18.335657 13383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:18.336949 13383 log.cc:824] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:13:18 dist-test-slave-f5h6 krb5kdc[13303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:18.340484 13383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c: No bootstrap required, opened a new log
I20260927 15:13:18.346700 13386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:18.324736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34090 (local address 127.10.33.126:37243)
0927 15:13:18.325584 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:18.325597 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:18.325639 (+ 42us) server_negotiation.cc:407] Connection header received
0927 15:13:18.327315 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:18.327339 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:18.327773 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:18.328133 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:18.329957 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:18.331225 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:18.332901 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:18.333608 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:18.333741 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:18.338438 (+ 4697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:18.338450 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:18.338473 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:18.338514 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:18.342278 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:18.342967 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:18.342976 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:18.342985 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:18.343192 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:18.343506 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:18.343511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:18.343514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:18.344212 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:18.344517 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:18.344837 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 15:13:18.345252 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":377,"threads_started":1}
I20260927 15:13:18.349952 13383 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3177e49c4624f728737775fdae6688c" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 37243 } }
I20260927 15:13:18.350303 13383 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:18.350397 13383 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3177e49c4624f728737775fdae6688c, State: Initialized, Role: FOLLOWER
I20260927 15:13:18.350670 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:13:18.351023 13383 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3177e49c4624f728737775fdae6688c" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 37243 } }
I20260927 15:13:18.351274 13383 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:13:18.351388 13383 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:13:18.351547 13383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:18.354592 13383 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3177e49c4624f728737775fdae6688c" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 37243 } }
I20260927 15:13:18.355180 13383 leader_election.cc:302] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3177e49c4624f728737775fdae6688c; no voters:
I20260927 15:13:18.355786 13383 leader_election.cc:288] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:13:18.355895 13388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:18.356935 13388 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c [term 1 LEADER]: Becoming Leader. State: Replica: c3177e49c4624f728737775fdae6688c, State: Running, Role: LEADER
I20260927 15:13:18.357448 13388 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3177e49c4624f728737775fdae6688c" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 37243 } }
I20260927 15:13:18.357960 13383 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:13:18.360239 13390 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3177e49c4624f728737775fdae6688c. Latest consensus state: current_term: 1 leader_uuid: "c3177e49c4624f728737775fdae6688c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3177e49c4624f728737775fdae6688c" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 37243 } } }
I20260927 15:13:18.360265 13389 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3177e49c4624f728737775fdae6688c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3177e49c4624f728737775fdae6688c" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 37243 } } }
I20260927 15:13:18.360651 13390 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:18.360651 13389 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:18.362206 13394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:13:18.368335 13394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:13:18.380254 13394 catalog_manager.cc:1347] Generated new cluster ID: 15f87a27e2784494af5908445fe31665
I20260927 15:13:18.380367 13394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:13:18.408550 13386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:18.399547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34100 (local address 127.10.33.126:37243)
0927 15:13:18.399766 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:18.399772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:18.400372 (+ 600us) server_negotiation.cc:407] Connection header received
0927 15:13:18.400552 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:18.400558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:18.400642 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:18.400746 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:18.401961 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:18.403016 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:18.404831 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:18.405439 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:18.405546 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:18.406766 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:18.406772 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:18.406777 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:13:18.407614 (+ 837us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:18.407685 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:13:18.407711 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:18.407813 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:18.407965 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 15:13:18.408059 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:13:18.414352 13394 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:13:18.416172 13394 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:13:18.429129 13394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3177e49c4624f728737775fdae6688c: Generated new TSK 0
I20260927 15:13:18.430305 13394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:13:18.444815 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 13316
I20260927 15:13:18.446255 13378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:18.537550 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 13316
2026-09-27T15:13:18Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1835 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:13:18 dist-test-slave-f5h6 krb5kdc[13417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:13:18 dist-test-slave-f5h6 krb5kdc[13417](info): set up 2 sockets
Sep 27 15:13:18 dist-test-slave-f5h6 krb5kdc[13417](info): commencing operation
krb5kdc: starting...
W20260927 15:13:19.538049 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:13:19 dist-test-slave-f5h6 krb5kdc[13417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:13:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:13:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:13:19.738123 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:46541
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:46541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:20.075937 13430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:20.076257 13430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:20.076385 13430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:20.084076 13430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:13:20.084214 13430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:20.084263 13430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:20.084336 13430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:13:20.084411 13430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:13:20.094460 13430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:46541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:46541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:20.095894 13430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:20.097167 13430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:20.109793 13435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:20.109822 13438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:20.109843 13436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:20.110623 13430 server_base.cc:1034] running on GCE node
I20260927 15:13:20.111461 13430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:20.113003 13430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:20.114248 13430 hybrid_clock.cc:648] HybridClock initialized: now 1790522000114183 us; error 46 us; skew 500 ppm
I20260927 15:13:20.114646 13430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:20 dist-test-slave-f5h6 krb5kdc[13417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522000, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:20.118150 13430 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:13:20.119742 13430 webserver.cc:460] Webserver started at http://127.10.33.126:45861/ using document root <none> and password file <none>
I20260927 15:13:20.120412 13430 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:20.120513 13430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:20.120783 13430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:20.123894 13430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "7bf75081a7ca4df184c516f459028bb5"
format_stamp: "Formatted at 2026-09-27 15:13:20 on dist-test-slave-f5h6"
server_key: "d4153e4f15a69f079d0e7c8da00c49c5"
server_key_iv: "138185a38a00b26aab36270e8c4dffb0"
server_key_version: "encryptionkey@0"
I20260927 15:13:20.124663 13430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "7bf75081a7ca4df184c516f459028bb5"
format_stamp: "Formatted at 2026-09-27 15:13:20 on dist-test-slave-f5h6"
server_key: "d4153e4f15a69f079d0e7c8da00c49c5"
server_key_iv: "138185a38a00b26aab36270e8c4dffb0"
server_key_version: "encryptionkey@0"
I20260927 15:13:20.130535 13430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 15:13:20.134152 13445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:20.135628 13430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:13:20.135834 13430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "7bf75081a7ca4df184c516f459028bb5"
format_stamp: "Formatted at 2026-09-27 15:13:20 on dist-test-slave-f5h6"
server_key: "d4153e4f15a69f079d0e7c8da00c49c5"
server_key_iv: "138185a38a00b26aab36270e8c4dffb0"
server_key_version: "encryptionkey@0"
I20260927 15:13:20.136031 13430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:20.175844 13430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:20.180379 13430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:20.180651 13430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:20.197986 13430 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:46541
I20260927 15:13:20.198032 13496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:46541 every 8 connection(s)
I20260927 15:13:20.199395 13430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:13:20.203089 13497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:20.207612 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 13430
I20260927 15:13:20.207844 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:13:20.208439 10373 external_mini_cluster.cc:1428] Setting key fe3f14653f8cb52db72456a78a2663ef
I20260927 15:13:20.214993 13497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5: Bootstrap starting.
I20260927 15:13:20.218623 13497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:20.219645 13497 log.cc:824] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:20.222671 13497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5: No bootstrap required, opened a new log
Sep 27 15:13:20 dist-test-slave-f5h6 krb5kdc[13417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:20.231674 13497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf75081a7ca4df184c516f459028bb5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46541 } }
I20260927 15:13:20.232045 13497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:20.232143 13497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf75081a7ca4df184c516f459028bb5, State: Initialized, Role: FOLLOWER
I20260927 15:13:20.232336 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:20.210517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56364 (local address 127.10.33.126:46541)
0927 15:13:20.211402 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:20.211418 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:20.211476 (+ 58us) server_negotiation.cc:407] Connection header received
0927 15:13:20.213790 (+ 2314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:20.213820 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:20.214382 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:20.214833 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:20.216135 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:20.218111 (+ 1976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:20.219468 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:20.220467 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:20.220602 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:20.224760 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:20.224770 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:20.224799 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:20.224843 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:20.227988 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:20.228824 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:20.228831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:20.228838 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:20.229032 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:20.229440 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:20.229443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:20.229446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:20.230141 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:20.230379 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:20.230811 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 15:13:20.231196 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":365,"threads_started":1}
I20260927 15:13:20.232750 13497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf75081a7ca4df184c516f459028bb5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46541 } }
I20260927 15:13:20.233034 13497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:13:20.233166 13497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:13:20.233325 13497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:20.235888 13497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf75081a7ca4df184c516f459028bb5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46541 } }
I20260927 15:13:20.236408 13497 leader_election.cc:302] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bf75081a7ca4df184c516f459028bb5; no voters:
I20260927 15:13:20.237043 13497 leader_election.cc:288] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:13:20.237191 13502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:20.239377 13497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:13:20.244055 13502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5 [term 1 LEADER]: Becoming Leader. State: Replica: 7bf75081a7ca4df184c516f459028bb5, State: Running, Role: LEADER
I20260927 15:13:20.244629 13502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf75081a7ca4df184c516f459028bb5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46541 } }
I20260927 15:13:20.248075 13504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bf75081a7ca4df184c516f459028bb5. Latest consensus state: current_term: 1 leader_uuid: "7bf75081a7ca4df184c516f459028bb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf75081a7ca4df184c516f459028bb5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46541 } } }
I20260927 15:13:20.248512 13504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:20.248065 13503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bf75081a7ca4df184c516f459028bb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf75081a7ca4df184c516f459028bb5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46541 } } }
I20260927 15:13:20.249385 13503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:20.251667 13514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:13:20.256460 13514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:13:20.265754 13514 catalog_manager.cc:1347] Generated new cluster ID: 58ad69d9bf244c609e18e611043152e8
I20260927 15:13:20.265913 13514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:13:20.289620 13514 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:13:20.291646 13514 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:13:20.299942 13514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bf75081a7ca4df184c516f459028bb5: Generated new TSK 0
I20260927 15:13:20.301221 13514 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.33.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
I20260927 15:13:20.354866 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:46541
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:20.684916 13523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:20.685251 13523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:20.685389 13523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:20.693085 13523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:20.693228 13523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:20.693397 13523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:13:20.703555 13523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:46541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:20.705073 13523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:20.706692 13523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:20.720016 13528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:20.719992 13531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:20.719956 13529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:20.720875 13523 server_base.cc:1034] running on GCE node
I20260927 15:13:20.721498 13523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:20.722523 13523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:20.723769 13523 hybrid_clock.cc:648] HybridClock initialized: now 1790522000723719 us; error 43 us; skew 500 ppm
I20260927 15:13:20.724103 13523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:20 dist-test-slave-f5h6 krb5kdc[13417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522000, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:20.731914 13523 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:13:20.733441 13523 webserver.cc:460] Webserver started at http://127.10.33.65:46263/ using document root <none> and password file <none>
I20260927 15:13:20.734184 13523 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:20.734284 13523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:20.734546 13523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:20.737406 13523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "ac29831fa8814c8ebbd5d20d3599e86f"
format_stamp: "Formatted at 2026-09-27 15:13:20 on dist-test-slave-f5h6"
server_key: "af15dfd4d715e9833d87735949eccbaa"
server_key_iv: "2db179af957ad398fbd32564691d445c"
server_key_version: "encryptionkey@0"
I20260927 15:13:20.738200 13523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "ac29831fa8814c8ebbd5d20d3599e86f"
format_stamp: "Formatted at 2026-09-27 15:13:20 on dist-test-slave-f5h6"
server_key: "af15dfd4d715e9833d87735949eccbaa"
server_key_iv: "2db179af957ad398fbd32564691d445c"
server_key_version: "encryptionkey@0"
I20260927 15:13:20.743551 13523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:13:20.747215 13538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:20.748651 13523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:13:20.748843 13523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "ac29831fa8814c8ebbd5d20d3599e86f"
format_stamp: "Formatted at 2026-09-27 15:13:20 on dist-test-slave-f5h6"
server_key: "af15dfd4d715e9833d87735949eccbaa"
server_key_iv: "2db179af957ad398fbd32564691d445c"
server_key_version: "encryptionkey@0"
I20260927 15:13:20.749045 13523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:20.784292 13523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:20.788717 13523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:20.789008 13523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:20.790019 13523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:20.791416 13523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:20.791491 13523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:20.791595 13523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:20.791649 13523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:20.822988 13523 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:46569
I20260927 15:13:20.823060 13650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:46569 every 8 connection(s)
I20260927 15:13:20.824234 13523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:13:20.825614 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 13523
I20260927 15:13:20.825802 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:13:20.826467 10373 external_mini_cluster.cc:1428] Setting key 853ff5fefd3fc3a917ad597363c6e180
Sep 27 15:13:20 dist-test-slave-f5h6 krb5kdc[13417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522000, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.33.66@KRBTEST.COM; defaulting to no policy
I20260927 15:13:20.844841 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:20.826854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:35725 (local address 127.10.33.126:46541)
0927 15:13:20.827311 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:20.827317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:20.828394 (+ 1077us) server_negotiation.cc:407] Connection header received
0927 15:13:20.830161 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:20.830166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:20.830264 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:20.830357 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:20.831922 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:20.832705 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:20.834323 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:20.834900 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:20.835021 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:20.838483 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:20.838489 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:20.838494 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:20.838534 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:20.841710 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:20.842561 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:20.842567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:20.842571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:20.842713 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:20.843174 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:20.843179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:20.843182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:20.843524 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:20.843660 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:20.844403 (+ 743us) server_negotiation.cc:299] Negotiation successful
0927 15:13:20.844584 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":216,"threads_started":1}
I20260927 15:13:20.846055 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:20.827461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:46541 (local address 127.10.33.65:35725)
0927 15:13:20.828192 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:20.828291 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:20.829908 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:20.830620 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:20.830629 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:20.831169 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:20.831753 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:20.831771 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:20.832886 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:20.832890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:20.834175 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:20.834187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:20.835108 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:20.835113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:20.835304 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:20.835968 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:20.835994 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:20.838247 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:20.841921 (+ 3674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:20.841927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:20.841942 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:20.842430 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:20.842808 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:20.842842 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:20.842845 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:20.843033 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:20.843678 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:20.843685 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:20.844018 (+ 333us) client_negotiation.cc:769] Sending connection context
0927 15:13:20.844268 (+ 250us) client_negotiation.cc:240] Negotiation successful
0927 15:13:20.844810 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":300,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.66@KRBTEST.COM" created.
I20260927 15:13:20.847589 13651 heartbeater.cc:347] Connected to a master server at 127.10.33.126:46541
I20260927 15:13:20.847911 13651 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:20.848583 13651 heartbeater.cc:511] Master 127.10.33.126:46541 requested a full tablet report, sending...
I20260927 15:13:20.850678 13462 ts_manager.cc:194] Registered new tserver with Master: ac29831fa8814c8ebbd5d20d3599e86f (127.10.33.65:46569)
I20260927 15:13:20.852669 13462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:35725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
I20260927 15:13:20.870198 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:46541
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:21.201463 13657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:21.201743 13657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:21.201881 13657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:21.209316 13657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:21.209432 13657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:21.209576 13657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:13:21.219683 13657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:46541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:21.221052 13657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:21.222430 13657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:21.235401 13662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:21.235515 13663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:21.235513 13665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:21.236339 13657 server_base.cc:1034] running on GCE node
I20260927 15:13:21.236931 13657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:21.237905 13657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:21.239104 13657 hybrid_clock.cc:648] HybridClock initialized: now 1790522001239042 us; error 58 us; skew 500 ppm
I20260927 15:13:21.239423 13657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:21 dist-test-slave-f5h6 krb5kdc[13417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522001, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:21.242848 13657 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:13:21.244442 13657 webserver.cc:460] Webserver started at http://127.10.33.66:38905/ using document root <none> and password file <none>
I20260927 15:13:21.245131 13657 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:21.245234 13657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:21.245488 13657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:21.248399 13657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "f738f50cbc5f43689766302cc8c611cb"
format_stamp: "Formatted at 2026-09-27 15:13:21 on dist-test-slave-f5h6"
server_key: "ca2fe8d6efc4ffa4d42c633687692817"
server_key_iv: "efa3a83c29681286e5f3172854c828cf"
server_key_version: "encryptionkey@0"
I20260927 15:13:21.249189 13657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "f738f50cbc5f43689766302cc8c611cb"
format_stamp: "Formatted at 2026-09-27 15:13:21 on dist-test-slave-f5h6"
server_key: "ca2fe8d6efc4ffa4d42c633687692817"
server_key_iv: "efa3a83c29681286e5f3172854c828cf"
server_key_version: "encryptionkey@0"
I20260927 15:13:21.254329 13657 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 15:13:21.257840 13672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:21.259493 13657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:13:21.259642 13657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "f738f50cbc5f43689766302cc8c611cb"
format_stamp: "Formatted at 2026-09-27 15:13:21 on dist-test-slave-f5h6"
server_key: "ca2fe8d6efc4ffa4d42c633687692817"
server_key_iv: "efa3a83c29681286e5f3172854c828cf"
server_key_version: "encryptionkey@0"
I20260927 15:13:21.259827 13657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:21.275537 13657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:21.279817 13657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:21.280071 13657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:21.281013 13657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:21.282449 13657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:21.282522 13657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:21.282609 13657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:21.282655 13657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:21.315863 13657 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:46453
I20260927 15:13:21.315892 13784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:46453 every 8 connection(s)
I20260927 15:13:21.317111 13657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:13:21.320679 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 13657
I20260927 15:13:21.320865 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:13:21.321576 10373 external_mini_cluster.cc:1428] Setting key e005c2fcc5eed58efe06491cad43023d
Sep 27 15:13:21 dist-test-slave-f5h6 krb5kdc[13417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522001, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:21.339962 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:21.319683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:45897 (local address 127.10.33.126:46541)
0927 15:13:21.319859 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:21.319863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:21.321279 (+ 1416us) server_negotiation.cc:407] Connection header received
0927 15:13:21.323354 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:21.323360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:21.323436 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:21.323513 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:21.325145 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:21.326261 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:21.327951 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:21.328549 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:21.328687 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:21.333269 (+ 4582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:21.333276 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:21.333281 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:21.333378 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:21.336242 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:21.337281 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:21.337285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:21.337288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:21.337446 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:21.337981 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:21.337985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:21.337987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:21.338282 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:21.338424 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:21.339392 (+ 968us) server_negotiation.cc:299] Negotiation successful
0927 15:13:21.339610 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.10.33.67@KRBTEST.COM; defaulting to no policy
I20260927 15:13:21.341200 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:21.320150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:46541 (local address 127.10.33.66:45897)
0927 15:13:21.321087 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:21.321144 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:21.323068 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:21.323770 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:21.323779 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:21.324293 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:21.324944 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:21.324963 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:21.326456 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:21.326461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:21.327776 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:21.327790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:21.328835 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:21.328853 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:21.329197 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:21.330070 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:21.330100 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:21.332939 (+ 2839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:21.336450 (+ 3511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:21.336461 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:21.336483 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:21.337131 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:21.337566 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:21.337571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:21.337575 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:21.337851 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:21.338427 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:21.338436 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:21.338918 (+ 482us) client_negotiation.cc:769] Sending connection context
0927 15:13:21.339260 (+ 342us) client_negotiation.cc:240] Negotiation successful
0927 15:13:21.339849 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":320,"threads_started":1}
I20260927 15:13:21.342938 13785 heartbeater.cc:347] Connected to a master server at 127.10.33.126:46541
I20260927 15:13:21.343262 13785 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.67@KRBTEST.COM" created.
I20260927 15:13:21.344149 13785 heartbeater.cc:511] Master 127.10.33.126:46541 requested a full tablet report, sending...
I20260927 15:13:21.345891 13462 ts_manager.cc:194] Registered new tserver with Master: f738f50cbc5f43689766302cc8c611cb (127.10.33.66:46453)
I20260927 15:13:21.346741 13462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:45897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
I20260927 15:13:21.370512 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:46541
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:21.694924 13790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:21.695333 13790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:21.695521 13790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:21.703202 13790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:21.703342 13790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:21.703512 13790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:13:21.713554 13790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:46541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:21.714931 13790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:21.716223 13790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:21.729724 13798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:21.729808 13795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:21.729799 13796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:21.730515 13790 server_base.cc:1034] running on GCE node
I20260927 15:13:21.731151 13790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:21.732133 13790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:21.733363 13790 hybrid_clock.cc:648] HybridClock initialized: now 1790522001733291 us; error 59 us; skew 500 ppm
I20260927 15:13:21.733681 13790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:21 dist-test-slave-f5h6 krb5kdc[13417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522001, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:21.736692 13790 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:13:21.738219 13790 webserver.cc:460] Webserver started at http://127.10.33.67:44167/ using document root <none> and password file <none>
I20260927 15:13:21.738907 13790 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:21.739025 13790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:21.739353 13790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:21.742620 13790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "2458b0c82e2b45228a9cb86f074859c8"
format_stamp: "Formatted at 2026-09-27 15:13:21 on dist-test-slave-f5h6"
server_key: "86525a12e1868f481f64328ca7c8af55"
server_key_iv: "a3133c96230b5bcb1cf106d1a02b6618"
server_key_version: "encryptionkey@0"
I20260927 15:13:21.743527 13790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "2458b0c82e2b45228a9cb86f074859c8"
format_stamp: "Formatted at 2026-09-27 15:13:21 on dist-test-slave-f5h6"
server_key: "86525a12e1868f481f64328ca7c8af55"
server_key_iv: "a3133c96230b5bcb1cf106d1a02b6618"
server_key_version: "encryptionkey@0"
I20260927 15:13:21.749094 13790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:13:21.752705 13805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:21.754264 13790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:13:21.754426 13790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "2458b0c82e2b45228a9cb86f074859c8"
format_stamp: "Formatted at 2026-09-27 15:13:21 on dist-test-slave-f5h6"
server_key: "86525a12e1868f481f64328ca7c8af55"
server_key_iv: "a3133c96230b5bcb1cf106d1a02b6618"
server_key_version: "encryptionkey@0"
I20260927 15:13:21.754607 13790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:21.777325 13790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:21.781718 13790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:21.782050 13790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:21.783109 13790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:21.784606 13790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:21.784700 13790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:21.784830 13790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:21.784883 13790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:21.814867 13790 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:38247
I20260927 15:13:21.814913 13917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:38247 every 8 connection(s)
I20260927 15:13:21.816118 13790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:13:21.819532 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 13790
I20260927 15:13:21.819720 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:13:21.820286 10373 external_mini_cluster.cc:1428] Setting key ac787038cbaca562354e18a68de2857f
Sep 27 15:13:21 dist-test-slave-f5h6 krb5kdc[13417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522001, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:21.837049 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:21.818745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:57625 (local address 127.10.33.126:46541)
0927 15:13:21.818940 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:21.818945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:21.820346 (+ 1401us) server_negotiation.cc:407] Connection header received
0927 15:13:21.822525 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:21.822531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:21.822647 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:21.822755 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:21.824507 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:21.825610 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:21.827295 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:21.827916 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:21.828048 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:21.831306 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:21.831317 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:21.831320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:21.831353 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:21.833590 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:21.834503 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:21.834508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:21.834511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:21.834664 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:21.835217 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:21.835222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:21.835225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:21.835552 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:21.835689 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:21.836578 (+ 889us) server_negotiation.cc:299] Negotiation successful
0927 15:13:21.836793 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 15:13:21.837949 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:21.819290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:46541 (local address 127.10.33.67:57625)
0927 15:13:21.820120 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:21.820169 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:21.822180 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:21.823030 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:21.823042 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:21.823725 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:21.824311 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:21.824330 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:21.825794 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:21.825799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:21.827136 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:21.827149 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:21.828102 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:21.828107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:21.828325 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:21.828968 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:21.828992 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:21.831096 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:21.833727 (+ 2631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:21.833735 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:21.833752 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:21.834337 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:21.834844 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:21.834850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:21.834854 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:21.835095 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:21.835702 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:21.835709 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:21.836183 (+ 474us) client_negotiation.cc:769] Sending connection context
0927 15:13:21.836438 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 15:13:21.836834 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":309,"threads_started":1}
I20260927 15:13:21.839293 13918 heartbeater.cc:347] Connected to a master server at 127.10.33.126:46541
I20260927 15:13:21.839605 13918 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:21.840374 13918 heartbeater.cc:511] Master 127.10.33.126:46541 requested a full tablet report, sending...
I20260927 15:13:21.841725 13462 ts_manager.cc:194] Registered new tserver with Master: 2458b0c82e2b45228a9cb86f074859c8 (127.10.33.67:38247)
I20260927 15:13:21.842599 13462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:57625
I20260927 15:13:21.848357 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:13:21.855734 13651 heartbeater.cc:503] Master 127.10.33.126:46541 was elected leader, sending a full tablet report...
Sep 27 15:13:21 dist-test-slave-f5h6 krb5kdc[13417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522001, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 15:13:21 dist-test-slave-f5h6 krb5kdc[13417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522001, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:21.891947 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:21.878307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56378 (local address 127.10.33.126:46541)
0927 15:13:21.878460 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:21.878489 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:21.878882 (+ 393us) server_negotiation.cc:407] Connection header received
0927 15:13:21.878999 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:21.879003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:21.879076 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:21.879172 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:21.880521 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:21.881311 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:21.882679 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:21.883164 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:21.883282 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:21.887196 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:21.887202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:21.887206 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:21.887247 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:21.889668 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:21.890361 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:21.890386 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:21.890388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:21.890528 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:21.890828 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:21.890831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:21.890833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:21.891152 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:21.891333 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:21.891559 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 15:13:21.891724 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 15:13:21.892532 13462 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56378
I20260927 15:13:21.894642 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 13523
Sep 27 15:13:21 dist-test-slave-f5h6 krb5kdc[13417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522001, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.33.65@KRBTEST.COM
I20260927 15:13:21.910089 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:21.895738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43830 (local address 127.10.33.65:46569)
0927 15:13:21.896322 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:21.896329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:21.896351 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:13:21.896411 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:21.896417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:21.896624 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:21.896758 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:21.897927 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:21.898912 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:21.900156 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:21.900736 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:21.900857 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:21.904660 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:21.904671 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:21.904685 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:21.904718 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:21.907341 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:21.908017 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:21.908024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:21.908027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:21.908148 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:21.908492 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:21.908496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:21.908498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:21.908837 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:21.909130 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:21.909453 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 15:13:21.909706 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":294,"threads_started":1}
W20260927 15:13:21.911386 13646 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43830
W20260927 15:13:21.912492 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 13523: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:13:21.912685 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 13523
I20260927 15:13:21.925365 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 13657
Sep 27 15:13:21 dist-test-slave-f5h6 krb5kdc[13417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522001, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.33.66@KRBTEST.COM
I20260927 15:13:21.942288 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:21.926280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36394 (local address 127.10.33.66:46453)
0927 15:13:21.926842 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:21.926849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:21.926876 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:13:21.926935 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:21.926942 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:21.927156 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:21.927300 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:21.928545 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:21.929884 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:21.931277 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:21.931871 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:21.931986 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:21.936288 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:21.936309 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:21.936332 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:21.936382 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:21.939281 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:21.940086 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:21.940094 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:21.940097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:21.940240 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:21.940681 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:21.940685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:21.940687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:21.941190 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:21.941431 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:21.941703 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 15:13:21.942009 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":290,"threads_started":1}
W20260927 15:13:21.943564 13780 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36394
W20260927 15:13:21.944739 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 13657: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:13:21.944970 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 13657
I20260927 15:13:21.957970 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 13790
Sep 27 15:13:21 dist-test-slave-f5h6 krb5kdc[13417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522001, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.33.67@KRBTEST.COM
I20260927 15:13:21.974685 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:21.959075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49848 (local address 127.10.33.67:38247)
0927 15:13:21.959970 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:21.959977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:21.960010 (+ 33us) server_negotiation.cc:407] Connection header received
0927 15:13:21.960071 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:21.960078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:21.960291 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:21.960458 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:21.961814 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:21.962877 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:21.964068 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:21.964635 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:21.964748 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:21.969016 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:21.969029 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:21.969046 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:21.969089 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:21.971938 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:21.972611 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:21.972618 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:21.972621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:21.972744 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:21.973106 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:21.973110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:21.973112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:21.973465 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:21.973767 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:21.974084 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 15:13:21.974355 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":379,"threads_started":1}
W20260927 15:13:21.976001 13913 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49848
W20260927 15:13:21.977110 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 13790: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:13:21.977270 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 13790
I20260927 15:13:21.989253 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 13430
I20260927 15:13:21.990610 13492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:22.096045 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 13430
2026-09-27T15:13:22Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3562 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:13:22 dist-test-slave-f5h6 krb5kdc[13935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:13:22 dist-test-slave-f5h6 krb5kdc[13935](info): set up 2 sockets
Sep 27 15:13:22 dist-test-slave-f5h6 krb5kdc[13935](info): commencing operation
krb5kdc: starting...
W20260927 15:13:23.106202 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:13:23 dist-test-slave-f5h6 krb5kdc[13935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:13:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:13:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:13:23.303603 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:44429
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:44429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:23.629416 13948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:23.629722 13948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:23.629812 13948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:23.637390 13948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:13:23.637539 13948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:23.637620 13948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:23.637702 13948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:13:23.637760 13948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:13:23.647667 13948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:44429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:44429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:23.648926 13948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:23.650185 13948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:23.661443 13953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:23.661504 13956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:23.661484 13954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:23.662163 13948 server_base.cc:1034] running on GCE node
I20260927 15:13:23.662928 13948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:23.664398 13948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:23.665647 13948 hybrid_clock.cc:648] HybridClock initialized: now 1790522003665585 us; error 49 us; skew 500 ppm
I20260927 15:13:23.666049 13948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:23 dist-test-slave-f5h6 krb5kdc[13935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522003, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:23.669379 13948 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:13:23.670987 13948 webserver.cc:460] Webserver started at http://127.10.33.126:38861/ using document root <none> and password file <none>
I20260927 15:13:23.671691 13948 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:23.671811 13948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:23.672116 13948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:23.675173 13948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "8c2cdad9ef4f4d57a123ccde5f105f1f"
format_stamp: "Formatted at 2026-09-27 15:13:23 on dist-test-slave-f5h6"
server_key: "d6053cba3cf104e4748333447a194dc2"
server_key_iv: "8d75e833000b695265bef28331bd1166"
server_key_version: "encryptionkey@0"
I20260927 15:13:23.675956 13948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "8c2cdad9ef4f4d57a123ccde5f105f1f"
format_stamp: "Formatted at 2026-09-27 15:13:23 on dist-test-slave-f5h6"
server_key: "d6053cba3cf104e4748333447a194dc2"
server_key_iv: "8d75e833000b695265bef28331bd1166"
server_key_version: "encryptionkey@0"
I20260927 15:13:23.681375 13948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:13:23.685178 13963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:23.686703 13948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:13:23.686947 13948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "8c2cdad9ef4f4d57a123ccde5f105f1f"
format_stamp: "Formatted at 2026-09-27 15:13:23 on dist-test-slave-f5h6"
server_key: "d6053cba3cf104e4748333447a194dc2"
server_key_iv: "8d75e833000b695265bef28331bd1166"
server_key_version: "encryptionkey@0"
I20260927 15:13:23.687165 13948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:23.729504 13948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:23.734210 13948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:23.734508 13948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:23.751888 13948 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:44429
I20260927 15:13:23.751895 14014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:44429 every 8 connection(s)
I20260927 15:13:23.753237 13948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:13:23.757304 14015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:23.763605 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 13948
I20260927 15:13:23.763899 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:13:23.764758 10373 external_mini_cluster.cc:1428] Setting key fc2f169016db2ece5ea9196e503367e8
I20260927 15:13:23.769824 14015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f: Bootstrap starting.
I20260927 15:13:23.773190 14015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:23.774353 14015 log.cc:824] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:23.777321 14015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f: No bootstrap required, opened a new log
Sep 27 15:13:23 dist-test-slave-f5h6 krb5kdc[13935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:23.785111 14015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2cdad9ef4f4d57a123ccde5f105f1f" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44429 } }
I20260927 15:13:23.785501 14015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:23.785596 14015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c2cdad9ef4f4d57a123ccde5f105f1f, State: Initialized, Role: FOLLOWER
I20260927 15:13:23.786374 14015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2cdad9ef4f4d57a123ccde5f105f1f" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44429 } }
I20260927 15:13:23.786666 14015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:13:23.786808 14015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:13:23.786988 14015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:23.788141 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:23.767205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55116 (local address 127.10.33.126:44429)
0927 15:13:23.768085 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:23.768097 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:23.768137 (+ 40us) server_negotiation.cc:407] Connection header received
0927 15:13:23.769828 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:23.769852 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:23.770324 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:23.770686 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:23.772085 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:23.773400 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:23.774701 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:23.775753 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:23.775927 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:23.780117 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:23.780125 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:23.780142 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:23.780183 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:23.783211 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:23.783924 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:23.783931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:23.783938 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:23.784124 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:23.784489 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:23.784493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:23.784495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:23.785073 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:23.785359 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:23.785984 (+ 625us) server_negotiation.cc:299] Negotiation successful
0927 15:13:23.786452 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":406,"threads_started":1}
I20260927 15:13:23.790272 14015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2cdad9ef4f4d57a123ccde5f105f1f" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44429 } }
I20260927 15:13:23.790814 14015 leader_election.cc:302] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c2cdad9ef4f4d57a123ccde5f105f1f; no voters:
I20260927 15:13:23.791605 14015 leader_election.cc:288] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:13:23.791728 14020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:23.792410 14020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f [term 1 LEADER]: Becoming Leader. State: Replica: 8c2cdad9ef4f4d57a123ccde5f105f1f, State: Running, Role: LEADER
I20260927 15:13:23.792949 14020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2cdad9ef4f4d57a123ccde5f105f1f" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44429 } }
I20260927 15:13:23.793830 14015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:13:23.796684 14021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c2cdad9ef4f4d57a123ccde5f105f1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2cdad9ef4f4d57a123ccde5f105f1f" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44429 } } }
I20260927 15:13:23.797128 14021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:23.796718 14022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c2cdad9ef4f4d57a123ccde5f105f1f. Latest consensus state: current_term: 1 leader_uuid: "8c2cdad9ef4f4d57a123ccde5f105f1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2cdad9ef4f4d57a123ccde5f105f1f" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44429 } } }
I20260927 15:13:23.797384 14022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:23.802682 14025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:13:23.807416 14025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:13:23.816540 14025 catalog_manager.cc:1347] Generated new cluster ID: 924f1ee987e340f6952427c06462fce2
I20260927 15:13:23.816650 14025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:13:23.829087 14025 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:13:23.830452 14025 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:13:23.840757 14025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c2cdad9ef4f4d57a123ccde5f105f1f: Generated new TSK 0
I20260927 15:13:23.841719 14025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.33.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
I20260927 15:13:23.898401 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:44429
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:24.244978 14041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:24.245287 14041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:24.245404 14041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:24.252978 14041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:24.253113 14041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:24.253260 14041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:13:24.263427 14041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:44429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:24.264775 14041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:24.266073 14041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:24.278946 14046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:24.278896 14047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:24.278882 14049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:24.279666 14041 server_base.cc:1034] running on GCE node
I20260927 15:13:24.280277 14041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:24.281304 14041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:24.282588 14041 hybrid_clock.cc:648] HybridClock initialized: now 1790522004282507 us; error 64 us; skew 500 ppm
I20260927 15:13:24.282970 14041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:24 dist-test-slave-f5h6 krb5kdc[13935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522004, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:24.286280 14041 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:13:24.287722 14041 webserver.cc:460] Webserver started at http://127.10.33.65:40299/ using document root <none> and password file <none>
I20260927 15:13:24.288424 14041 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:24.288538 14041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:24.288908 14041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:24.292125 14041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "371a3e430dd74a6a9fe4e0c0b88e587c"
format_stamp: "Formatted at 2026-09-27 15:13:24 on dist-test-slave-f5h6"
server_key: "b38292fe229192bb204fca0c95527508"
server_key_iv: "4af2e0d3c98e0962c96cb0d63af56f06"
server_key_version: "encryptionkey@0"
I20260927 15:13:24.292910 14041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "371a3e430dd74a6a9fe4e0c0b88e587c"
format_stamp: "Formatted at 2026-09-27 15:13:24 on dist-test-slave-f5h6"
server_key: "b38292fe229192bb204fca0c95527508"
server_key_iv: "4af2e0d3c98e0962c96cb0d63af56f06"
server_key_version: "encryptionkey@0"
I20260927 15:13:24.298166 14041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 15:13:24.301674 14056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:24.303267 14041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:13:24.303437 14041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "371a3e430dd74a6a9fe4e0c0b88e587c"
format_stamp: "Formatted at 2026-09-27 15:13:24 on dist-test-slave-f5h6"
server_key: "b38292fe229192bb204fca0c95527508"
server_key_iv: "4af2e0d3c98e0962c96cb0d63af56f06"
server_key_version: "encryptionkey@0"
I20260927 15:13:24.303629 14041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:24.335675 14041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:24.340072 14041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:24.340390 14041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:24.341349 14041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:24.342724 14041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:24.342798 14041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:24.342903 14041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:24.342953 14041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:24.371893 14041 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:38671
I20260927 15:13:24.371934 14168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:38671 every 8 connection(s)
I20260927 15:13:24.373178 14041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:13:24.380753 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 14041
I20260927 15:13:24.381029 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:13:24.381728 10373 external_mini_cluster.cc:1428] Setting key 99a8b8d408bbb8910a65e026bf785f22
Sep 27 15:13:24 dist-test-slave-f5h6 krb5kdc[13935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522004, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:24.394292 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:24.375787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:37213 (local address 127.10.33.126:44429)
0927 15:13:24.376234 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:24.376239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:24.377261 (+ 1022us) server_negotiation.cc:407] Connection header received
0927 15:13:24.379177 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:24.379182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:24.379268 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:24.379359 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:24.381128 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:24.381911 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:24.383633 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:24.384069 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:24.384183 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:24.388031 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:24.388037 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:24.388040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:24.388073 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:24.391091 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:24.391888 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:24.391894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:24.391896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:24.392029 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:24.392466 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:24.392470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:24.392472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:24.392739 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:24.392880 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:24.393620 (+ 740us) server_negotiation.cc:299] Negotiation successful
0927 15:13:24.393915 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":240,"threads_started":1}
I20260927 15:13:24.395329 14172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:24.376294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:44429 (local address 127.10.33.65:37213)
0927 15:13:24.377083 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:24.377124 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:24.378876 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:24.379628 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:24.379637 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:24.380198 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:24.380809 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:24.380828 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:24.382080 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:24.382085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:24.383485 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:24.383499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:24.384303 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:24.384312 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:24.384560 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:24.385314 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:24.385338 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:24.387763 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:24.391265 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:24.391292 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:24.391308 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:24.391759 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:24.392142 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:24.392147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:24.392151 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:24.392353 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:24.392867 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:24.392876 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:24.393246 (+ 370us) client_negotiation.cc:769] Sending connection context
0927 15:13:24.393506 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 15:13:24.394100 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":279,"threads_started":1}
I20260927 15:13:24.396930 14169 heartbeater.cc:347] Connected to a master server at 127.10.33.126:44429
I20260927 15:13:24.397215 14169 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:24.397975 14169 heartbeater.cc:511] Master 127.10.33.126:44429 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.33.66@KRBTEST.COM; defaulting to no policy
I20260927 15:13:24.399947 13980 ts_manager.cc:194] Registered new tserver with Master: 371a3e430dd74a6a9fe4e0c0b88e587c (127.10.33.65:38671)
I20260927 15:13:24.401641 13980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:37213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
I20260927 15:13:24.425387 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:44429
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:24.754366 14175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:24.754642 14175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:24.754757 14175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:24.762079 14175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:24.762188 14175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:24.762343 14175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:13:24.772022 14175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:44429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:24.773255 14175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:24.774534 14175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:24.787170 14181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:24.787232 14180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:24.787209 14183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:24.787861 14175 server_base.cc:1034] running on GCE node
I20260927 15:13:24.788537 14175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:24.789537 14175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:24.790733 14175 hybrid_clock.cc:648] HybridClock initialized: now 1790522004790684 us; error 35 us; skew 500 ppm
I20260927 15:13:24.791093 14175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:24 dist-test-slave-f5h6 krb5kdc[13935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522004, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:24.794346 14175 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:13:24.796216 14175 webserver.cc:460] Webserver started at http://127.10.33.66:42565/ using document root <none> and password file <none>
I20260927 15:13:24.796984 14175 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:24.797127 14175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:24.797451 14175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:24.800866 14175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "10927eb179a249b4a83504094e32c0b7"
format_stamp: "Formatted at 2026-09-27 15:13:24 on dist-test-slave-f5h6"
server_key: "c836567830fe58e49c5e28d329551288"
server_key_iv: "69176cf1dd354134ac012b0482a9eda6"
server_key_version: "encryptionkey@0"
I20260927 15:13:24.801797 14175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "10927eb179a249b4a83504094e32c0b7"
format_stamp: "Formatted at 2026-09-27 15:13:24 on dist-test-slave-f5h6"
server_key: "c836567830fe58e49c5e28d329551288"
server_key_iv: "69176cf1dd354134ac012b0482a9eda6"
server_key_version: "encryptionkey@0"
I20260927 15:13:24.807466 14175 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 15:13:24.811092 14190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:24.812592 14175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:13:24.812767 14175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "10927eb179a249b4a83504094e32c0b7"
format_stamp: "Formatted at 2026-09-27 15:13:24 on dist-test-slave-f5h6"
server_key: "c836567830fe58e49c5e28d329551288"
server_key_iv: "69176cf1dd354134ac012b0482a9eda6"
server_key_version: "encryptionkey@0"
I20260927 15:13:24.812968 14175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:24.848379 14175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:24.852505 14175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:24.852768 14175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:24.853694 14175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:24.855032 14175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:24.855108 14175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:24.855237 14175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:24.855288 14175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:24.884367 14175 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:34079
I20260927 15:13:24.884420 14302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:34079 every 8 connection(s)
I20260927 15:13:24.885655 14175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:13:24.894502 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 14175
I20260927 15:13:24.894706 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:13:24.895303 10373 external_mini_cluster.cc:1428] Setting key e21c7c521ad472ceb67402f9037f38a2
Sep 27 15:13:24 dist-test-slave-f5h6 krb5kdc[13935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522004, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:24.907498 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:24.888216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:56579 (local address 127.10.33.126:44429)
0927 15:13:24.888425 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:24.888432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:24.889693 (+ 1261us) server_negotiation.cc:407] Connection header received
0927 15:13:24.891666 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:24.891672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:24.891775 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:24.891870 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:24.893464 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:24.894327 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:24.896579 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:24.897253 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:24.897380 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:24.901658 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:24.901663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:24.901667 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:24.901702 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:24.904508 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:24.905216 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:24.905221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:24.905223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:24.905344 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:24.905746 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:24.905750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:24.905752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:24.906068 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:24.906210 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:24.906935 (+ 725us) server_negotiation.cc:299] Negotiation successful
0927 15:13:24.907173 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:13:24.908649 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:24.888759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:44429 (local address 127.10.33.66:56579)
0927 15:13:24.889513 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:24.889554 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:24.891345 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:24.892111 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:24.892119 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:24.892685 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:24.893275 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:24.893293 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:24.894487 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:24.894493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:24.896437 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:24.896451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:24.897488 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:24.897496 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:24.897805 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:24.898576 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:24.898609 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:24.901342 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:24.904641 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:24.904648 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:24.904663 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:24.905086 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:24.905449 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:24.905454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:24.905457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:24.905639 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:24.906184 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:24.906192 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:24.906579 (+ 387us) client_negotiation.cc:769] Sending connection context
0927 15:13:24.906826 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 15:13:24.907475 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":299,"threads_started":1}
I20260927 15:13:24.910341 14303 heartbeater.cc:347] Connected to a master server at 127.10.33.126:44429
I20260927 15:13:24.910624 14303 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:24.911346 14303 heartbeater.cc:511] Master 127.10.33.126:44429 requested a full tablet report, sending...
I20260927 15:13:24.912904 13980 ts_manager.cc:194] Registered new tserver with Master: 10927eb179a249b4a83504094e32c0b7 (127.10.33.66:34079)
WARNING: no policy specified for kudu/127.10.33.67@KRBTEST.COM; defaulting to no policy
I20260927 15:13:24.914068 13980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:56579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
I20260927 15:13:24.938053 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:44429
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:25.267590 14308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:25.267851 14308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:25.267951 14308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:25.275276 14308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:25.275394 14308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:25.275511 14308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:13:25.285491 14308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:44429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:25.286859 14308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:25.288199 14308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:25.301252 14316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:25.301308 14313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:25.301308 14314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:25.301785 14308 server_base.cc:1034] running on GCE node
I20260927 15:13:25.302587 14308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:25.303594 14308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:25.304888 14308 hybrid_clock.cc:648] HybridClock initialized: now 1790522005304812 us; error 58 us; skew 500 ppm
I20260927 15:13:25.305199 14308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:25 dist-test-slave-f5h6 krb5kdc[13935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522005, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:25.308525 14308 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:13:25.310009 14308 webserver.cc:460] Webserver started at http://127.10.33.67:45747/ using document root <none> and password file <none>
I20260927 15:13:25.310760 14308 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:25.310863 14308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:25.311157 14308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:25.314487 14308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "92144a98f9524cb2a8447415733e1181"
format_stamp: "Formatted at 2026-09-27 15:13:25 on dist-test-slave-f5h6"
server_key: "1fb1a84f0f87f3c406385e0ff67f4eca"
server_key_iv: "e6dc62f9897dcd828091c1bd0c785dca"
server_key_version: "encryptionkey@0"
I20260927 15:13:25.315295 14308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "92144a98f9524cb2a8447415733e1181"
format_stamp: "Formatted at 2026-09-27 15:13:25 on dist-test-slave-f5h6"
server_key: "1fb1a84f0f87f3c406385e0ff67f4eca"
server_key_iv: "e6dc62f9897dcd828091c1bd0c785dca"
server_key_version: "encryptionkey@0"
I20260927 15:13:25.320935 14308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260927 15:13:25.324702 14323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:25.326195 14308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:13:25.326370 14308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "92144a98f9524cb2a8447415733e1181"
format_stamp: "Formatted at 2026-09-27 15:13:25 on dist-test-slave-f5h6"
server_key: "1fb1a84f0f87f3c406385e0ff67f4eca"
server_key_iv: "e6dc62f9897dcd828091c1bd0c785dca"
server_key_version: "encryptionkey@0"
I20260927 15:13:25.326560 14308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:25.348649 14308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:25.352998 14308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:25.353284 14308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:25.354326 14308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:25.355654 14308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:25.355727 14308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:25.355823 14308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:25.355880 14308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:25.387005 14308 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:46107
I20260927 15:13:25.387072 14435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:46107 every 8 connection(s)
I20260927 15:13:25.388271 14308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:13:25.396875 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 14308
I20260927 15:13:25.397069 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:13:25.397563 10373 external_mini_cluster.cc:1428] Setting key 359b826525add9ee2c127425dc5564e0
Sep 27 15:13:25 dist-test-slave-f5h6 krb5kdc[13935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522005, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:25.404879 14169 heartbeater.cc:503] Master 127.10.33.126:44429 was elected leader, sending a full tablet report...
I20260927 15:13:25.410501 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:25.390855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:42071 (local address 127.10.33.126:44429)
0927 15:13:25.391078 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:25.391084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:25.392517 (+ 1433us) server_negotiation.cc:407] Connection header received
0927 15:13:25.394735 (+ 2218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:25.394740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:25.394835 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:25.394930 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:25.396481 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:25.397571 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:25.399476 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:25.400045 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:25.400173 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:25.403854 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:25.403860 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:25.403863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:25.403905 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:25.407161 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:25.408047 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:25.408052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:25.408054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:25.408190 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:25.408718 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:25.408722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:25.408724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:25.409075 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:25.409237 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:25.410079 (+ 842us) server_negotiation.cc:299] Negotiation successful
0927 15:13:25.410250 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 15:13:25.411532 14438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:25.391472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:44429 (local address 127.10.33.67:42071)
0927 15:13:25.392315 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:25.392375 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:25.394432 (+ 2057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:25.395134 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:25.395143 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:25.395693 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:25.396276 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:25.396298 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:25.397755 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:25.397761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:25.399314 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:25.399327 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:25.400263 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:25.400268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:25.400558 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:25.401232 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:25.401256 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:25.403553 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:25.407313 (+ 3760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:25.407321 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:25.407338 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:25.407899 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:25.408334 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:25.408338 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:25.408341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:25.408573 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:25.409228 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:25.409236 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:25.409622 (+ 386us) client_negotiation.cc:769] Sending connection context
0927 15:13:25.409960 (+ 338us) client_negotiation.cc:240] Negotiation successful
0927 15:13:25.410389 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":327,"threads_started":1}
I20260927 15:13:25.413110 14436 heartbeater.cc:347] Connected to a master server at 127.10.33.126:44429
I20260927 15:13:25.413432 14436 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:25.414222 14436 heartbeater.cc:511] Master 127.10.33.126:44429 requested a full tablet report, sending...
I20260927 15:13:25.415659 13980 ts_manager.cc:194] Registered new tserver with Master: 92144a98f9524cb2a8447415733e1181 (127.10.33.67:46107)
I20260927 15:13:25.416512 13980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:42071
I20260927 15:13:25.426009 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:13:25 dist-test-slave-f5h6 krb5kdc[13935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:13:25 dist-test-slave-f5h6 krb5kdc[13935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.65@KRBTEST.COM
I20260927 15:13:25.471372 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:25.456999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39758 (local address 127.10.33.65:38671)
0927 15:13:25.457566 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:25.457573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:25.457599 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:13:25.457657 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:25.457663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:25.457943 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:25.458080 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:25.459463 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:25.460493 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:25.461716 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:25.462361 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:25.462459 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:25.466209 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:25.466221 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:25.466236 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:25.466270 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:25.468742 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:25.469355 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:25.469362 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:25.469364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:25.469490 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:25.469833 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:25.469837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:25.469839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:25.470201 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:25.470479 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:25.470768 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 15:13:25.471033 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":278,"threads_started":1}
Sep 27 15:13:25 dist-test-slave-f5h6 krb5kdc[13935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:25.488878 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:25.475569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55124 (local address 127.10.33.126:44429)
0927 15:13:25.475765 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:25.475771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:25.475894 (+ 123us) server_negotiation.cc:407] Connection header received
0927 15:13:25.476055 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:25.476060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:25.476121 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:25.476209 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:25.477430 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:25.478184 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:25.479349 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:25.479751 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:25.479855 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:25.483576 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:25.483581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:25.483585 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:25.483623 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:25.486396 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:25.487115 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:25.487119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:25.487122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:25.487261 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:25.487602 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:25.487606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:25.487608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:25.488001 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:25.488198 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:25.488428 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 15:13:25.488598 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 15:13:25.489513 13980 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55124
I20260927 15:13:25.490535 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 14041
I20260927 15:13:25.503579 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:25.491583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39766 (local address 127.10.33.65:38671)
0927 15:13:25.491754 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:25.491760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:25.492336 (+ 576us) server_negotiation.cc:407] Connection header received
0927 15:13:25.492434 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:25.492438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:25.492532 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:25.492613 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:25.493820 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:25.494607 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:25.495891 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:25.496301 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:25.496401 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:25.498640 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:25.498645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:25.498649 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:25.498685 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:25.501152 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:25.501774 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:25.501778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:25.501781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:25.501932 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:25.502259 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:25.502264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:25.502266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:25.502589 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:25.502770 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:25.503100 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 15:13:25.503314 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:13:25.504129 14164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:25.585525 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 14041
I20260927 15:13:25.598778 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 14175
Sep 27 15:13:25 dist-test-slave-f5h6 krb5kdc[13935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.66@KRBTEST.COM
I20260927 15:13:25.615347 14448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:25.600088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34180 (local address 127.10.33.66:34079)
0927 15:13:25.600670 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:25.600678 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:25.600704 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:13:25.600764 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:25.600771 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:25.600979 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:25.601122 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:25.602640 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:25.603728 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:25.605130 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:25.605710 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:25.605821 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:25.609809 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:25.609826 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:25.609842 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:25.609921 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:25.612537 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:25.613248 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:25.613256 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:25.613259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:25.613411 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:25.613698 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:25.613701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:25.613703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:25.614114 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:25.614364 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:25.614735 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 15:13:25.614947 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":286,"threads_started":1}
I20260927 15:13:25.616654 14298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:25.698045 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 14175
I20260927 15:13:25.710482 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 14308
Sep 27 15:13:25 dist-test-slave-f5h6 krb5kdc[13935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.67@KRBTEST.COM
I20260927 15:13:25.726722 14450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:25.711431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.10.33.67:46107)
0927 15:13:25.712065 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:25.712072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:25.712100 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:13:25.712158 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:25.712164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:25.712370 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:25.712539 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:25.713751 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:25.714822 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:25.716134 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:25.716703 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:25.716835 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:25.721035 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:25.721050 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:25.721065 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:25.721111 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:25.724117 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:25.724881 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:25.724889 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:25.724891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:25.725024 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:25.725339 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:25.725343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:25.725345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:25.725710 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:25.726033 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:25.726224 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 15:13:25.726440 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":333,"threads_started":1}
I20260927 15:13:25.727963 14431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:25.816829 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 14308
I20260927 15:13:25.828953 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 13948
I20260927 15:13:25.830180 14010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:25.930733 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 13948
2026-09-27T15:13:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3835 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:13:26 dist-test-slave-f5h6 krb5kdc[14455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:13:26 dist-test-slave-f5h6 krb5kdc[14455](info): set up 2 sockets
Sep 27 15:13:26 dist-test-slave-f5h6 krb5kdc[14455](info): commencing operation
krb5kdc: starting...
W20260927 15:13:26.950160 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:13:27 dist-test-slave-f5h6 krb5kdc[14455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:13:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:13:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:13:27.150683 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:42181
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:32955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:42181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:27.482388 14468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:27.482712 14468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:27.482813 14468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:27.490368 14468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:13:27.490497 14468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:27.490548 14468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:27.490633 14468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:13:27.490903 14468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:13:27.501572 14468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:32955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:42181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:42181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:27.503027 14468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:27.504429 14468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:27.516382 14473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:27.516443 14474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:27.516494 14476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:27.517324 14468 server_base.cc:1034] running on GCE node
I20260927 15:13:27.518360 14468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:27.519994 14468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:27.521245 14468 hybrid_clock.cc:648] HybridClock initialized: now 1790522007521148 us; error 93 us; skew 500 ppm
I20260927 15:13:27.521716 14468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:27 dist-test-slave-f5h6 krb5kdc[14455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522007, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:27.525200 14468 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:13:27.526830 14468 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:27.526955 14468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:27.527281 14468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:27.531014 14468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "94b02e69470941c3a0c114e9a9798ccc"
format_stamp: "Formatted at 2026-09-27 15:13:27 on dist-test-slave-f5h6"
server_key: "88557b1e6492cf83e0fb8039bcf2ec8b"
server_key_iv: "83f8326dc3d62462c1ba444321a98ca5"
server_key_version: "encryptionkey@0"
I20260927 15:13:27.531971 14468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "94b02e69470941c3a0c114e9a9798ccc"
format_stamp: "Formatted at 2026-09-27 15:13:27 on dist-test-slave-f5h6"
server_key: "88557b1e6492cf83e0fb8039bcf2ec8b"
server_key_iv: "83f8326dc3d62462c1ba444321a98ca5"
server_key_version: "encryptionkey@0"
I20260927 15:13:27.537551 14468 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 15:13:27.541498 14482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:27.543069 14468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:13:27.543260 14468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "94b02e69470941c3a0c114e9a9798ccc"
format_stamp: "Formatted at 2026-09-27 15:13:27 on dist-test-slave-f5h6"
server_key: "88557b1e6492cf83e0fb8039bcf2ec8b"
server_key_iv: "83f8326dc3d62462c1ba444321a98ca5"
server_key_version: "encryptionkey@0"
I20260927 15:13:27.543497 14468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:27.595227 14468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:27.600566 14468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:27.600857 14468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:27.622684 14468 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:42181
I20260927 15:13:27.622699 14533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:42181 every 8 connection(s)
I20260927 15:13:27.623620 14468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:13:27.627836 14534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:27.631856 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 14468
I20260927 15:13:27.632148 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:13:27.633068 10373 external_mini_cluster.cc:1428] Setting key a27f51344eb8e5a9cad1aa1396d8c6a1
I20260927 15:13:27.644374 14534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc: Bootstrap starting.
I20260927 15:13:27.649387 14534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:27.650899 14534 log.cc:824] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:13:27 dist-test-slave-f5h6 krb5kdc[14455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:27.654964 14534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc: No bootstrap required, opened a new log
I20260927 15:13:27.665689 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:27.635483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55530 (local address 127.10.33.126:42181)
0927 15:13:27.636322 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:27.636339 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:27.636387 (+ 48us) server_negotiation.cc:407] Connection header received
0927 15:13:27.638985 (+ 2598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:27.639017 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:27.639521 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:27.639946 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:27.643896 (+ 3950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:27.645699 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:27.648146 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:27.649277 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:27.649769 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:27.655365 (+ 5596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:27.655380 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:27.655401 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:27.655458 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:27.659027 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:27.660565 (+ 1538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:27.660575 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:27.660584 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:27.660824 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:27.661375 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:27.661379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:27.661382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:27.662375 (+ 993us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:27.662633 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:27.663401 (+ 768us) server_negotiation.cc:299] Negotiation successful
0927 15:13:27.663823 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":381,"threads_started":1}
I20260927 15:13:27.667932 14534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b02e69470941c3a0c114e9a9798ccc" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42181 } }
I20260927 15:13:27.668460 14534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:27.668568 14534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94b02e69470941c3a0c114e9a9798ccc, State: Initialized, Role: FOLLOWER
I20260927 15:13:27.669238 14534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b02e69470941c3a0c114e9a9798ccc" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42181 } }
I20260927 15:13:27.669477 14534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:13:27.669596 14534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:13:27.669742 14534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:27.672196 14534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b02e69470941c3a0c114e9a9798ccc" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42181 } }
I20260927 15:13:27.672747 14534 leader_election.cc:302] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94b02e69470941c3a0c114e9a9798ccc; no voters:
I20260927 15:13:27.673368 14534 leader_election.cc:288] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:13:27.673522 14539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:27.674234 14539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc [term 1 LEADER]: Becoming Leader. State: Replica: 94b02e69470941c3a0c114e9a9798ccc, State: Running, Role: LEADER
I20260927 15:13:27.674823 14539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b02e69470941c3a0c114e9a9798ccc" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42181 } }
I20260927 15:13:27.675513 14534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:13:27.678946 14541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94b02e69470941c3a0c114e9a9798ccc. Latest consensus state: current_term: 1 leader_uuid: "94b02e69470941c3a0c114e9a9798ccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b02e69470941c3a0c114e9a9798ccc" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42181 } } }
I20260927 15:13:27.679636 14541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:27.681502 14540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94b02e69470941c3a0c114e9a9798ccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b02e69470941c3a0c114e9a9798ccc" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42181 } } }
I20260927 15:13:27.682040 14540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:27.685662 14545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:13:27.691402 14545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:13:27.706476 14545 catalog_manager.cc:1347] Generated new cluster ID: aaa2a87cec61497eacf204fbd8039bea
I20260927 15:13:27.706609 14545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:13:27.746393 14545 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:13:27.748193 14545 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:13:27.755630 14545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc: Generated new TSK 0
I20260927 15:13:27.756701 14545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.33.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
I20260927 15:13:27.829815 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:42181
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:32955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:28.187770 14560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:28.188077 14560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:28.188176 14560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:28.196081 14560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:28.196230 14560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:28.196352 14560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:13:28.206784 14560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:32955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:42181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:28.208199 14560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:28.209794 14560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:28.223603 14568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:28.223642 14566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:28.223677 14565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:28.224578 14560 server_base.cc:1034] running on GCE node
I20260927 15:13:28.225172 14560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:28.226178 14560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:28.227378 14560 hybrid_clock.cc:648] HybridClock initialized: now 1790522008227326 us; error 47 us; skew 500 ppm
I20260927 15:13:28.227694 14560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:28 dist-test-slave-f5h6 krb5kdc[14455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522008, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:28.230893 14560 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:13:28.232100 14560 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:28.232200 14560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:28.232497 14560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:28.235463 14560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "1651e92a6d51470f9a346a053e2e8e02"
format_stamp: "Formatted at 2026-09-27 15:13:28 on dist-test-slave-f5h6"
server_key: "a8e1fb595c3cf7fcb855e990b6995854"
server_key_iv: "b3714d00e837ba81ac4d17e6d6bed871"
server_key_version: "encryptionkey@0"
I20260927 15:13:28.236268 14560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "1651e92a6d51470f9a346a053e2e8e02"
format_stamp: "Formatted at 2026-09-27 15:13:28 on dist-test-slave-f5h6"
server_key: "a8e1fb595c3cf7fcb855e990b6995854"
server_key_iv: "b3714d00e837ba81ac4d17e6d6bed871"
server_key_version: "encryptionkey@0"
I20260927 15:13:28.241570 14560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:13:28.245570 14574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:28.247368 14560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:13:28.247562 14560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "1651e92a6d51470f9a346a053e2e8e02"
format_stamp: "Formatted at 2026-09-27 15:13:28 on dist-test-slave-f5h6"
server_key: "a8e1fb595c3cf7fcb855e990b6995854"
server_key_iv: "b3714d00e837ba81ac4d17e6d6bed871"
server_key_version: "encryptionkey@0"
I20260927 15:13:28.247781 14560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:28.283164 14560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:28.287881 14560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:28.288147 14560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:28.289228 14560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:28.290608 14560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:28.290702 14560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:28.290817 14560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:28.290892 14560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:28.322928 14560 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:38707
I20260927 15:13:28.322986 14686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:38707 every 8 connection(s)
I20260927 15:13:28.323758 14560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:13:28.330084 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 14560
I20260927 15:13:28.330298 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:13:28.330921 10373 external_mini_cluster.cc:1428] Setting key 82cbd1737616ddd6927fc3ba9cb3727e
Sep 27 15:13:28 dist-test-slave-f5h6 krb5kdc[14455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522008, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:28.343801 14689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:28.326306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:40107 (local address 127.10.33.126:42181)
0927 15:13:28.326766 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:28.326771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:28.327730 (+ 959us) server_negotiation.cc:407] Connection header received
0927 15:13:28.329570 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:28.329575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:28.329661 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:28.329753 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:28.331435 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:28.332277 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:28.333952 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:28.334533 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:28.334661 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:28.338126 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:28.338134 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:28.338140 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:28.338181 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:28.340624 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:28.341521 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:28.341526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:28.341529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:28.341650 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:28.342130 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:28.342133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:28.342136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:28.342396 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:28.342506 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:28.343356 (+ 850us) server_negotiation.cc:299] Negotiation successful
0927 15:13:28.343533 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":265,"threads_started":1}
I20260927 15:13:28.345155 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:28.326752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:42181 (local address 127.10.33.65:40107)
0927 15:13:28.327540 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:28.327587 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:28.329263 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:28.330034 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:28.330044 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:28.330646 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:28.331258 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:28.331277 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:28.332473 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:28.332478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:28.333777 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:28.333790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:28.334764 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:28.334770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:28.335047 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:28.335741 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:28.335766 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:28.337837 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:28.340768 (+ 2931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:28.340777 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:28.340797 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:28.341348 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:28.341748 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:28.341752 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:28.341756 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:28.342031 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:28.342504 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:28.342511 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:28.342969 (+ 458us) client_negotiation.cc:769] Sending connection context
0927 15:13:28.343274 (+ 305us) client_negotiation.cc:240] Negotiation successful
0927 15:13:28.343800 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":302,"threads_started":1}
I20260927 15:13:28.346712 14687 heartbeater.cc:347] Connected to a master server at 127.10.33.126:42181
I20260927 15:13:28.347064 14687 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:28.347930 14687 heartbeater.cc:511] Master 127.10.33.126:42181 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.33.66@KRBTEST.COM; defaulting to no policy
I20260927 15:13:28.350234 14499 ts_manager.cc:194] Registered new tserver with Master: 1651e92a6d51470f9a346a053e2e8e02 (127.10.33.65:38707)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.66@KRBTEST.COM" created.
I20260927 15:13:28.351934 14499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:40107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
I20260927 15:13:28.373222 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:42181
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:32955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:28.704931 14693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:28.705247 14693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:28.705387 14693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:28.713012 14693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:28.713189 14693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:28.713346 14693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:13:28.723598 14693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:32955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:42181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:28.725033 14693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:28.726398 14693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:28.739830 14698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:28.739801 14699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:28.740317 14701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:28.740727 14693 server_base.cc:1034] running on GCE node
I20260927 15:13:28.741408 14693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:28.742530 14693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:28.743762 14693 hybrid_clock.cc:648] HybridClock initialized: now 1790522008743699 us; error 60 us; skew 500 ppm
I20260927 15:13:28.744113 14693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:28 dist-test-slave-f5h6 krb5kdc[14455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522008, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:28.747295 14693 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:13:28.748594 14693 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:28.748703 14693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:28.749037 14693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:28.752115 14693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "cfac505369d148dcadc7ce02a4341994"
format_stamp: "Formatted at 2026-09-27 15:13:28 on dist-test-slave-f5h6"
server_key: "1142c0eac90bdaa3aa1e41c836338391"
server_key_iv: "f336af115ded76efcc7a2de6f64d0d6f"
server_key_version: "encryptionkey@0"
I20260927 15:13:28.752915 14693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "cfac505369d148dcadc7ce02a4341994"
format_stamp: "Formatted at 2026-09-27 15:13:28 on dist-test-slave-f5h6"
server_key: "1142c0eac90bdaa3aa1e41c836338391"
server_key_iv: "f336af115ded76efcc7a2de6f64d0d6f"
server_key_version: "encryptionkey@0"
I20260927 15:13:28.757920 14693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260927 15:13:28.761540 14707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:28.762979 14693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:13:28.763135 14693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "cfac505369d148dcadc7ce02a4341994"
format_stamp: "Formatted at 2026-09-27 15:13:28 on dist-test-slave-f5h6"
server_key: "1142c0eac90bdaa3aa1e41c836338391"
server_key_iv: "f336af115ded76efcc7a2de6f64d0d6f"
server_key_version: "encryptionkey@0"
I20260927 15:13:28.763337 14693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:28.776327 14693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:28.781069 14693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:28.781407 14693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:28.782603 14693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:28.784179 14693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:28.784286 14693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:28.784395 14693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:28.784461 14693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:28.814532 14693 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:40163
I20260927 15:13:28.814601 14819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:40163 every 8 connection(s)
I20260927 15:13:28.815601 14693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:13:28.822540 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 14693
I20260927 15:13:28.822784 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:13:28.823330 10373 external_mini_cluster.cc:1428] Setting key 3b68eac0e321f08980346be21c19a9bb
Sep 27 15:13:28 dist-test-slave-f5h6 krb5kdc[14455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522008, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:28.838894 14689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:28.818200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:35441 (local address 127.10.33.126:42181)
0927 15:13:28.818464 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:28.818469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:28.819694 (+ 1225us) server_negotiation.cc:407] Connection header received
0927 15:13:28.821640 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:28.821645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:28.821716 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:28.821802 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:28.823967 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:28.824867 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:28.827407 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:28.828032 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:28.828201 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:28.832540 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:28.832547 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:28.832552 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:28.832600 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:28.835655 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:28.836528 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:28.836533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:28.836537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:28.836675 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:28.837100 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:28.837105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:28.837107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:28.837436 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:28.837575 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:28.838337 (+ 762us) server_negotiation.cc:299] Negotiation successful
0927 15:13:28.838574 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260927 15:13:28.839841 14822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:28.818768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:42181 (local address 127.10.33.66:35441)
0927 15:13:28.819523 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:28.819570 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:28.821349 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:28.822063 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:28.822072 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:28.822854 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:28.823695 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:28.823723 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:28.825085 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:28.825089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:28.827100 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:28.827122 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:28.828364 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:28.828372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:28.828681 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:28.829497 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:28.829533 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:28.832184 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:28.835850 (+ 3666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:28.835858 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:28.835916 (+ 58us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:28.836389 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:28.836766 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:28.836769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:28.836771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:28.836988 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:28.837556 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:28.837563 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:28.837980 (+ 417us) client_negotiation.cc:769] Sending connection context
0927 15:13:28.838219 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 15:13:28.838622 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":295,"threads_started":1}
I20260927 15:13:28.841265 14820 heartbeater.cc:347] Connected to a master server at 127.10.33.126:42181
I20260927 15:13:28.841562 14820 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:28.842263 14820 heartbeater.cc:511] Master 127.10.33.126:42181 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.33.67@KRBTEST.COM; defaulting to no policy
I20260927 15:13:28.843986 14499 ts_manager.cc:194] Registered new tserver with Master: cfac505369d148dcadc7ce02a4341994 (127.10.33.66:40163)
I20260927 15:13:28.844818 14499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:35441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
I20260927 15:13:28.869153 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:42181
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:32955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:29.202214 14825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:29.202512 14825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:29.202641 14825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:29.210997 14825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:29.211160 14825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:29.211336 14825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:13:29.221612 14825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:32955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:42181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:29.223160 14825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:29.224524 14825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:29.236860 14833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:29.236949 14831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:29.236876 14830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:29.237582 14825 server_base.cc:1034] running on GCE node
I20260927 15:13:29.238190 14825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:29.239118 14825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:29.240345 14825 hybrid_clock.cc:648] HybridClock initialized: now 1790522009240276 us; error 63 us; skew 500 ppm
I20260927 15:13:29.240679 14825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:29 dist-test-slave-f5h6 krb5kdc[14455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522009, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:29.244415 14825 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:13:29.245661 14825 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:29.245766 14825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:29.246124 14825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:29.249253 14825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "8380ecc66ca14d8ba6164faaa97e3b4e"
format_stamp: "Formatted at 2026-09-27 15:13:29 on dist-test-slave-f5h6"
server_key: "321c41eb04fc8def0762bd47adf71c87"
server_key_iv: "4743ece7a3a3fe6a49aeb79fa5d1e02d"
server_key_version: "encryptionkey@0"
I20260927 15:13:29.250151 14825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "8380ecc66ca14d8ba6164faaa97e3b4e"
format_stamp: "Formatted at 2026-09-27 15:13:29 on dist-test-slave-f5h6"
server_key: "321c41eb04fc8def0762bd47adf71c87"
server_key_iv: "4743ece7a3a3fe6a49aeb79fa5d1e02d"
server_key_version: "encryptionkey@0"
I20260927 15:13:29.257460 14825 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260927 15:13:29.261516 14839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:29.263130 14825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 15:13:29.263346 14825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "8380ecc66ca14d8ba6164faaa97e3b4e"
format_stamp: "Formatted at 2026-09-27 15:13:29 on dist-test-slave-f5h6"
server_key: "321c41eb04fc8def0762bd47adf71c87"
server_key_iv: "4743ece7a3a3fe6a49aeb79fa5d1e02d"
server_key_version: "encryptionkey@0"
I20260927 15:13:29.263581 14825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:29.290124 14825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:29.295109 14825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:29.295387 14825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:29.296408 14825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:29.297897 14825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:29.297997 14825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:29.298110 14825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:29.298177 14825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:29.329986 14825 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:33865
I20260927 15:13:29.330057 14951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:33865 every 8 connection(s)
I20260927 15:13:29.330812 14825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:13:29.338692 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 14825
I20260927 15:13:29.338891 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:13:29.339423 10373 external_mini_cluster.cc:1428] Setting key 18366bc12ed6a7c52d48976d87dd36ad
Sep 27 15:13:29 dist-test-slave-f5h6 krb5kdc[14455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522009, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:29.352907 14689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:29.333113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:47045 (local address 127.10.33.126:42181)
0927 15:13:29.333281 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:29.333286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:29.334481 (+ 1195us) server_negotiation.cc:407] Connection header received
0927 15:13:29.336192 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:29.336196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:29.336264 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:29.336350 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:29.337829 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.338667 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.340787 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.341345 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.341451 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:29.344994 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:29.345000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:29.345003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:29.345038 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:29.347964 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:29.349353 (+ 1389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:29.349359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:29.349362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:29.349584 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:29.350382 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:29.350387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:29.350390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:29.350737 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:29.350887 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:29.351872 (+ 985us) server_negotiation.cc:299] Negotiation successful
0927 15:13:29.352245 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:13:29.353449 14954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:29.333532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:42181 (local address 127.10.33.67:47045)
0927 15:13:29.334313 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:29.334354 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:29.335935 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:29.336555 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:29.336563 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:29.337125 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:29.337665 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:29.337683 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.338826 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.338832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:29.340638 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:29.340653 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.341596 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.341602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:29.341797 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:29.342571 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:29.342603 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:29.344761 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:29.348219 (+ 3458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:29.348232 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:29.348254 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:29.349162 (+ 908us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:29.349902 (+ 740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:29.349909 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:29.349912 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:29.350205 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:29.350957 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:29.350967 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:29.351426 (+ 459us) client_negotiation.cc:769] Sending connection context
0927 15:13:29.351694 (+ 268us) client_negotiation.cc:240] Negotiation successful
0927 15:13:29.352213 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":379,"threads_started":1}
I20260927 15:13:29.354833 14687 heartbeater.cc:503] Master 127.10.33.126:42181 was elected leader, sending a full tablet report...
I20260927 15:13:29.355412 14952 heartbeater.cc:347] Connected to a master server at 127.10.33.126:42181
I20260927 15:13:29.355702 14952 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:29.356400 14952 heartbeater.cc:511] Master 127.10.33.126:42181 requested a full tablet report, sending...
I20260927 15:13:29.358171 14498 ts_manager.cc:194] Registered new tserver with Master: 8380ecc66ca14d8ba6164faaa97e3b4e (127.10.33.67:33865)
I20260927 15:13:29.359047 14498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:47045
I20260927 15:13:29.367624 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:13:29.386549 14689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:29.372913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55540 (local address 127.10.33.126:42181)
0927 15:13:29.373110 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:29.373115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:29.373614 (+ 499us) server_negotiation.cc:407] Connection header received
0927 15:13:29.373723 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:29.373727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:29.373792 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:29.373933 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:29.375639 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.376496 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.378081 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.378551 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.378663 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:29.381153 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:29.381158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:29.381161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:29.381203 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:29.383905 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:29.384663 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:29.384667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:29.384669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:29.384805 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:29.385253 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:29.385257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:29.385259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:29.385595 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:29.385785 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:29.386152 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 15:13:29.386310 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 15:13:29.391975 14498 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:13:29.399808 14498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:13:29.423179 14965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:29.412485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48854 (local address 127.10.33.65:38707)
0927 15:13:29.413290 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:29.413298 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:29.414326 (+ 1028us) server_negotiation.cc:407] Connection header received
0927 15:13:29.414643 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:29.414651 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:29.414901 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:29.415048 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:29.416536 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.418030 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.420536 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.421951 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.422093 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:29.422337 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:29.422675 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 15:13:29.422849 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":717,"threads_started":1}
I20260927 15:13:29.423234 14966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:29.412346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:38707 (local address 127.0.0.1:48854)
0927 15:13:29.414164 (+ 1818us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:29.414227 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:29.414424 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:29.415302 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:29.415306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:29.415342 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:29.415807 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:29.415823 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.418174 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.418179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:29.420362 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:29.420384 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.422153 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.422159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:29.422364 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:29.422436 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 15:13:29.422607 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 15:13:29.422818 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1552,"mutex_wait_us":56,"spinlock_wait_cycles":1408,"thread_start_us":1453,"threads_started":1}
I20260927 15:13:29.424777 14968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:29.414497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:33865 (local address 127.0.0.1:38460)
0927 15:13:29.415829 (+ 1332us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:29.415846 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:29.415952 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:29.416624 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:29.416628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:29.416651 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:29.416980 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:29.416990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.418977 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.418985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:29.421309 (+ 2324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:29.421335 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.423931 (+ 2596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.423938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:29.424113 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:29.424146 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:13:29.424237 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 15:13:29.424380 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1205,"mutex_wait_us":58,"thread_start_us":1055,"threads_started":1}
I20260927 15:13:29.425154 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:29.413053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35718 (local address 127.10.33.66:40163)
0927 15:13:29.413672 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:29.413680 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:29.414592 (+ 912us) server_negotiation.cc:407] Connection header received
0927 15:13:29.414692 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:29.414698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:29.414892 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:29.415056 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:29.416420 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.417933 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.421714 (+ 3781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.424082 (+ 2368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.424233 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:29.424421 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:29.424675 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 15:13:29.424862 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":379,"threads_started":1}
I20260927 15:13:29.425626 14969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:29.415004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38460 (local address 127.10.33.67:33865)
0927 15:13:29.416115 (+ 1111us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:29.416120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:29.416144 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:13:29.416214 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:29.416218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:29.416380 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:29.416656 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:29.417191 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.418718 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.421710 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.423715 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.423854 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:29.424092 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:29.425142 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0927 15:13:29.425325 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":969,"thread_start_us":843,"threads_started":1}
I20260927 15:13:29.425172 14964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:29.412781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:40163 (local address 127.0.0.1:35718)
0927 15:13:29.414164 (+ 1383us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:29.414227 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:29.414424 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:29.415295 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:29.415302 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:29.415350 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:29.416273 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:29.416281 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.418063 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.418068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:29.421309 (+ 3241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:29.421335 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.424312 (+ 2977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.424319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:29.424493 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:29.424530 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 15:13:29.424622 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 15:13:29.424774 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1221,"mutex_wait_us":167,"spinlock_wait_cycles":25728,"thread_start_us":921,"threads_started":1}
I20260927 15:13:29.427711 14622 tablet_service.cc:1467] Processing CreateTablet for tablet 472da491194a428496c40d8d529189f6 (DEFAULT_TABLE table=test-table [id=36ce8e26297c46e58bb1204626de3d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:29.428944 14622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472da491194a428496c40d8d529189f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:29.429414 14887 tablet_service.cc:1467] Processing CreateTablet for tablet 472da491194a428496c40d8d529189f6 (DEFAULT_TABLE table=test-table [id=36ce8e26297c46e58bb1204626de3d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:29.429520 14755 tablet_service.cc:1467] Processing CreateTablet for tablet 472da491194a428496c40d8d529189f6 (DEFAULT_TABLE table=test-table [id=36ce8e26297c46e58bb1204626de3d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:29.430836 14755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472da491194a428496c40d8d529189f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:29.430836 14887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472da491194a428496c40d8d529189f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:29.440482 14970 tablet_bootstrap.cc:492] T 472da491194a428496c40d8d529189f6 P 1651e92a6d51470f9a346a053e2e8e02: Bootstrap starting.
I20260927 15:13:29.443611 14970 tablet_bootstrap.cc:654] T 472da491194a428496c40d8d529189f6 P 1651e92a6d51470f9a346a053e2e8e02: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:29.444661 14970 log.cc:824] T 472da491194a428496c40d8d529189f6 P 1651e92a6d51470f9a346a053e2e8e02: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:29.445066 14971 tablet_bootstrap.cc:492] T 472da491194a428496c40d8d529189f6 P cfac505369d148dcadc7ce02a4341994: Bootstrap starting.
I20260927 15:13:29.445851 14972 tablet_bootstrap.cc:492] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e: Bootstrap starting.
I20260927 15:13:29.448817 14971 tablet_bootstrap.cc:654] T 472da491194a428496c40d8d529189f6 P cfac505369d148dcadc7ce02a4341994: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:29.449697 14972 tablet_bootstrap.cc:654] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:29.449787 14971 log.cc:824] T 472da491194a428496c40d8d529189f6 P cfac505369d148dcadc7ce02a4341994: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:29.450927 14972 log.cc:824] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:29.452733 14971 tablet_bootstrap.cc:492] T 472da491194a428496c40d8d529189f6 P cfac505369d148dcadc7ce02a4341994: No bootstrap required, opened a new log
I20260927 15:13:29.452998 14971 ts_tablet_manager.cc:1397] T 472da491194a428496c40d8d529189f6 P cfac505369d148dcadc7ce02a4341994: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 15:13:29.454182 14970 tablet_bootstrap.cc:492] T 472da491194a428496c40d8d529189f6 P 1651e92a6d51470f9a346a053e2e8e02: No bootstrap required, opened a new log
I20260927 15:13:29.454181 14972 tablet_bootstrap.cc:492] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e: No bootstrap required, opened a new log
I20260927 15:13:29.454450 14970 ts_tablet_manager.cc:1397] T 472da491194a428496c40d8d529189f6 P 1651e92a6d51470f9a346a053e2e8e02: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20260927 15:13:29.454450 14972 ts_tablet_manager.cc:1397] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260927 15:13:29.460255 14971 raft_consensus.cc:348] T 472da491194a428496c40d8d529189f6 P cfac505369d148dcadc7ce02a4341994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1651e92a6d51470f9a346a053e2e8e02" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 38707 } } peers { permanent_uuid: "8380ecc66ca14d8ba6164faaa97e3b4e" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 33865 } } peers { permanent_uuid: "cfac505369d148dcadc7ce02a4341994" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 40163 } }
I20260927 15:13:29.460635 14971 raft_consensus.cc:374] T 472da491194a428496c40d8d529189f6 P cfac505369d148dcadc7ce02a4341994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:29.460745 14971 raft_consensus.cc:729] T 472da491194a428496c40d8d529189f6 P cfac505369d148dcadc7ce02a4341994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfac505369d148dcadc7ce02a4341994, State: Initialized, Role: FOLLOWER
I20260927 15:13:29.461331 14971 consensus_queue.cc:260] T 472da491194a428496c40d8d529189f6 P cfac505369d148dcadc7ce02a4341994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1651e92a6d51470f9a346a053e2e8e02" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 38707 } } peers { permanent_uuid: "8380ecc66ca14d8ba6164faaa97e3b4e" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 33865 } } peers { permanent_uuid: "cfac505369d148dcadc7ce02a4341994" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 40163 } }
I20260927 15:13:29.462456 14820 heartbeater.cc:503] Master 127.10.33.126:42181 was elected leader, sending a full tablet report...
I20260927 15:13:29.462960 14971 ts_tablet_manager.cc:1428] T 472da491194a428496c40d8d529189f6 P cfac505369d148dcadc7ce02a4341994: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260927 15:13:29.464620 14972 raft_consensus.cc:348] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1651e92a6d51470f9a346a053e2e8e02" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 38707 } } peers { permanent_uuid: "8380ecc66ca14d8ba6164faaa97e3b4e" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 33865 } } peers { permanent_uuid: "cfac505369d148dcadc7ce02a4341994" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 40163 } }
I20260927 15:13:29.464969 14972 raft_consensus.cc:374] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:29.465060 14972 raft_consensus.cc:729] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8380ecc66ca14d8ba6164faaa97e3b4e, State: Initialized, Role: FOLLOWER
I20260927 15:13:29.465184 14970 raft_consensus.cc:348] T 472da491194a428496c40d8d529189f6 P 1651e92a6d51470f9a346a053e2e8e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1651e92a6d51470f9a346a053e2e8e02" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 38707 } } peers { permanent_uuid: "8380ecc66ca14d8ba6164faaa97e3b4e" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 33865 } } peers { permanent_uuid: "cfac505369d148dcadc7ce02a4341994" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 40163 } }
I20260927 15:13:29.465973 14970 raft_consensus.cc:374] T 472da491194a428496c40d8d529189f6 P 1651e92a6d51470f9a346a053e2e8e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:29.465735 14972 consensus_queue.cc:260] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1651e92a6d51470f9a346a053e2e8e02" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 38707 } } peers { permanent_uuid: "8380ecc66ca14d8ba6164faaa97e3b4e" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 33865 } } peers { permanent_uuid: "cfac505369d148dcadc7ce02a4341994" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 40163 } }
I20260927 15:13:29.466095 14970 raft_consensus.cc:729] T 472da491194a428496c40d8d529189f6 P 1651e92a6d51470f9a346a053e2e8e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1651e92a6d51470f9a346a053e2e8e02, State: Initialized, Role: FOLLOWER
I20260927 15:13:29.466806 14970 consensus_queue.cc:260] T 472da491194a428496c40d8d529189f6 P 1651e92a6d51470f9a346a053e2e8e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1651e92a6d51470f9a346a053e2e8e02" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 38707 } } peers { permanent_uuid: "8380ecc66ca14d8ba6164faaa97e3b4e" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 33865 } } peers { permanent_uuid: "cfac505369d148dcadc7ce02a4341994" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 40163 } }
I20260927 15:13:29.468669 14970 ts_tablet_manager.cc:1428] T 472da491194a428496c40d8d529189f6 P 1651e92a6d51470f9a346a053e2e8e02: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 15:13:29.469726 14952 heartbeater.cc:503] Master 127.10.33.126:42181 was elected leader, sending a full tablet report...
I20260927 15:13:29.470551 14972 ts_tablet_manager.cc:1428] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
W20260927 15:13:29.567524 14821 tablet.cc:2369] T 472da491194a428496c40d8d529189f6 P cfac505369d148dcadc7ce02a4341994: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:13:29.576078 14688 tablet.cc:2369] T 472da491194a428496c40d8d529189f6 P 1651e92a6d51470f9a346a053e2e8e02: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:13:29.582263 14953 tablet.cc:2369] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:13:29.653741 14977 raft_consensus.cc:482] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:13:29.654088 14977 raft_consensus.cc:504] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1651e92a6d51470f9a346a053e2e8e02" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 38707 } } peers { permanent_uuid: "8380ecc66ca14d8ba6164faaa97e3b4e" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 33865 } } peers { permanent_uuid: "cfac505369d148dcadc7ce02a4341994" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 40163 } }
I20260927 15:13:29.656131 14977 leader_election.cc:288] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1651e92a6d51470f9a346a053e2e8e02 (127.10.33.65:38707), cfac505369d148dcadc7ce02a4341994 (127.10.33.66:40163)
I20260927 15:13:29.663211 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:29.657059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:51951 (local address 127.10.33.66:40163)
0927 15:13:29.657227 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:29.657232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:29.657995 (+ 763us) server_negotiation.cc:407] Connection header received
0927 15:13:29.658137 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:29.658142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:29.658207 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:29.658276 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:29.658742 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.659524 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.661299 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.662326 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.662489 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:29.662563 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:29.662828 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 15:13:29.662955 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:13:29.663312 14979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:29.656441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:40163 (local address 127.10.33.67:51951)
0927 15:13:29.657879 (+ 1438us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:29.657904 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:29.658022 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:29.658302 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:29.658306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:29.658328 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:29.658615 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:29.658624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.659657 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.659662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:29.661149 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:29.661162 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.662504 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.662510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:29.662680 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:29.662713 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:13:29.662803 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 15:13:29.662950 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1301,"mutex_wait_us":803,"thread_start_us":327,"threads_started":1}
I20260927 15:13:29.664330 14775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472da491194a428496c40d8d529189f6" candidate_uuid: "8380ecc66ca14d8ba6164faaa97e3b4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfac505369d148dcadc7ce02a4341994" is_pre_election: true
I20260927 15:13:29.664888 14775 raft_consensus.cc:2454] T 472da491194a428496c40d8d529189f6 P cfac505369d148dcadc7ce02a4341994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8380ecc66ca14d8ba6164faaa97e3b4e in term 0.
I20260927 15:13:29.665298 14965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:29.656864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:42051 (local address 127.10.33.65:38707)
0927 15:13:29.657049 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:29.657055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:29.657072 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:13:29.657140 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:29.657144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:29.657215 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:29.657309 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:29.657901 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.659014 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.661590 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.663031 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.663202 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:29.663275 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:29.664781 (+ 1506us) server_negotiation.cc:299] Negotiation successful
0927 15:13:29.664939 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:13:29.665588 14842 leader_election.cc:302] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8380ecc66ca14d8ba6164faaa97e3b4e, cfac505369d148dcadc7ce02a4341994; no voters:
I20260927 15:13:29.665704 14954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:29.656430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:38707 (local address 127.10.33.67:42051)
0927 15:13:29.656622 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:29.656642 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:29.656760 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:29.657370 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:29.657376 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:29.657416 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:29.657675 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:29.657688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.659192 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.659198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:29.661376 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:29.661392 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.663701 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.663708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:29.663907 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:29.663944 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 15:13:29.664037 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 15:13:29.664191 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":1}
I20260927 15:13:29.666004 14977 raft_consensus.cc:2781] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:13:29.666153 14977 raft_consensus.cc:482] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:13:29.666249 14977 raft_consensus.cc:3037] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:29.666997 14642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472da491194a428496c40d8d529189f6" candidate_uuid: "8380ecc66ca14d8ba6164faaa97e3b4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1651e92a6d51470f9a346a053e2e8e02" is_pre_election: true
I20260927 15:13:29.667568 14642 raft_consensus.cc:2454] T 472da491194a428496c40d8d529189f6 P 1651e92a6d51470f9a346a053e2e8e02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8380ecc66ca14d8ba6164faaa97e3b4e in term 0.
I20260927 15:13:29.669085 14977 raft_consensus.cc:504] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1651e92a6d51470f9a346a053e2e8e02" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 38707 } } peers { permanent_uuid: "8380ecc66ca14d8ba6164faaa97e3b4e" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 33865 } } peers { permanent_uuid: "cfac505369d148dcadc7ce02a4341994" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 40163 } }
I20260927 15:13:29.669765 14977 leader_election.cc:288] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e [CANDIDATE]: Term 1 election: Requested vote from peers 1651e92a6d51470f9a346a053e2e8e02 (127.10.33.65:38707), cfac505369d148dcadc7ce02a4341994 (127.10.33.66:40163)
I20260927 15:13:29.670151 14642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472da491194a428496c40d8d529189f6" candidate_uuid: "8380ecc66ca14d8ba6164faaa97e3b4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1651e92a6d51470f9a346a053e2e8e02"
I20260927 15:13:29.670348 14642 raft_consensus.cc:3037] T 472da491194a428496c40d8d529189f6 P 1651e92a6d51470f9a346a053e2e8e02 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:29.670361 14775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472da491194a428496c40d8d529189f6" candidate_uuid: "8380ecc66ca14d8ba6164faaa97e3b4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfac505369d148dcadc7ce02a4341994"
I20260927 15:13:29.670547 14775 raft_consensus.cc:3037] T 472da491194a428496c40d8d529189f6 P cfac505369d148dcadc7ce02a4341994 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:29.673452 14642 raft_consensus.cc:2454] T 472da491194a428496c40d8d529189f6 P 1651e92a6d51470f9a346a053e2e8e02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8380ecc66ca14d8ba6164faaa97e3b4e in term 1.
I20260927 15:13:29.673453 14775 raft_consensus.cc:2454] T 472da491194a428496c40d8d529189f6 P cfac505369d148dcadc7ce02a4341994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8380ecc66ca14d8ba6164faaa97e3b4e in term 1.
I20260927 15:13:29.673944 14840 leader_election.cc:302] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1651e92a6d51470f9a346a053e2e8e02, 8380ecc66ca14d8ba6164faaa97e3b4e; no voters:
I20260927 15:13:29.674281 14977 raft_consensus.cc:2781] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:29.674728 14977 raft_consensus.cc:686] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e [term 1 LEADER]: Becoming Leader. State: Replica: 8380ecc66ca14d8ba6164faaa97e3b4e, State: Running, Role: LEADER
I20260927 15:13:29.675212 14977 consensus_queue.cc:237] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1651e92a6d51470f9a346a053e2e8e02" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 38707 } } peers { permanent_uuid: "8380ecc66ca14d8ba6164faaa97e3b4e" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 33865 } } peers { permanent_uuid: "cfac505369d148dcadc7ce02a4341994" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 40163 } }
I20260927 15:13:29.679944 14498 catalog_manager.cc:5495] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8380ecc66ca14d8ba6164faaa97e3b4e (127.10.33.67). New cstate: current_term: 1 leader_uuid: "8380ecc66ca14d8ba6164faaa97e3b4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1651e92a6d51470f9a346a053e2e8e02" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 38707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8380ecc66ca14d8ba6164faaa97e3b4e" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 33865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfac505369d148dcadc7ce02a4341994" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 40163 } health_report { overall_health: UNKNOWN } } }
I20260927 15:13:29.725474 14969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:29.719045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38474 (local address 127.10.33.67:33865)
0927 15:13:29.719230 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:29.719235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:29.719253 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:13:29.719552 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:29.719557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:29.719649 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:29.719783 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:13:29.720390 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.721290 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.722963 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.723396 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.723576 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:29.724203 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:13:29.724748 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:13:29.724869 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:29.725115 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 15:13:29.725200 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:13:29.734346 14775 raft_consensus.cc:1261] T 472da491194a428496c40d8d529189f6 P cfac505369d148dcadc7ce02a4341994 [term 1 FOLLOWER]: Refusing update from remote peer 8380ecc66ca14d8ba6164faaa97e3b4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:29.734346 14642 raft_consensus.cc:1261] T 472da491194a428496c40d8d529189f6 P 1651e92a6d51470f9a346a053e2e8e02 [term 1 FOLLOWER]: Refusing update from remote peer 8380ecc66ca14d8ba6164faaa97e3b4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:29.735291 14980 consensus_queue.cc:1035] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfac505369d148dcadc7ce02a4341994" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 40163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:29.735687 14977 consensus_queue.cc:1035] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1651e92a6d51470f9a346a053e2e8e02" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 38707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:29.745970 14983 mvcc.cc:205] Tried to move back new op lower bound from 7333978151861215232 to 7333978151633453056. Current Snapshot: MvccSnapshot[applied={T|T < 7333978151861215232}]
I20260927 15:13:29.749794 14984 mvcc.cc:205] Tried to move back new op lower bound from 7333978151861215232 to 7333978151633453056. Current Snapshot: MvccSnapshot[applied={T|T < 7333978151861215232}]
I20260927 15:13:29.762045 14499 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:29.762357 14499 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:29.766234 14499 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94b02e69470941c3a0c114e9a9798ccc: Sending DeleteTablet for 3 replicas of tablet 472da491194a428496c40d8d529189f6
I20260927 15:13:29.767711 14755 tablet_service.cc:1514] Processing DeleteTablet for tablet 472da491194a428496c40d8d529189f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:29 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:35718
I20260927 15:13:29.768496 14622 tablet_service.cc:1514] Processing DeleteTablet for tablet 472da491194a428496c40d8d529189f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:29 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:48854
I20260927 15:13:29.768608 14992 tablet_replica.cc:330] T 472da491194a428496c40d8d529189f6 P cfac505369d148dcadc7ce02a4341994: stopping tablet replica
I20260927 15:13:29.768683 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 14560
I20260927 15:13:29.769176 14992 raft_consensus.cc:2229] T 472da491194a428496c40d8d529189f6 P cfac505369d148dcadc7ce02a4341994 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:29.769770 14992 raft_consensus.cc:2258] T 472da491194a428496c40d8d529189f6 P cfac505369d148dcadc7ce02a4341994 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:29.769810 14887 tablet_service.cc:1514] Processing DeleteTablet for tablet 472da491194a428496c40d8d529189f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:29 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:38460
I20260927 15:13:29.772212 14992 ts_tablet_manager.cc:1905] T 472da491194a428496c40d8d529189f6 P cfac505369d148dcadc7ce02a4341994: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:29.778402 14994 tablet_replica.cc:330] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e: stopping tablet replica
I20260927 15:13:29.779037 14994 raft_consensus.cc:2229] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:13:29.779842 14994 raft_consensus.cc:2258] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:29.782733 14994 ts_tablet_manager.cc:1905] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:29.783089 14993 tablet_replica.cc:330] T 472da491194a428496c40d8d529189f6 P 1651e92a6d51470f9a346a053e2e8e02: stopping tablet replica
I20260927 15:13:29.783792 14993 raft_consensus.cc:2229] T 472da491194a428496c40d8d529189f6 P 1651e92a6d51470f9a346a053e2e8e02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:29.784318 14993 raft_consensus.cc:2258] T 472da491194a428496c40d8d529189f6 P 1651e92a6d51470f9a346a053e2e8e02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:29.785482 14992 ts_tablet_manager.cc:1918] T 472da491194a428496c40d8d529189f6 P cfac505369d148dcadc7ce02a4341994: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:13:29.785740 14992 log.cc:1196] T 472da491194a428496c40d8d529189f6 P cfac505369d148dcadc7ce02a4341994: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/472da491194a428496c40d8d529189f6
I20260927 15:13:29.786384 14992 ts_tablet_manager.cc:1939] T 472da491194a428496c40d8d529189f6 P cfac505369d148dcadc7ce02a4341994: Deleting consensus metadata
I20260927 15:13:29.787847 14993 ts_tablet_manager.cc:1905] T 472da491194a428496c40d8d529189f6 P 1651e92a6d51470f9a346a053e2e8e02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:29.788410 14485 catalog_manager.cc:4841] TS cfac505369d148dcadc7ce02a4341994 (127.10.33.66:40163): tablet 472da491194a428496c40d8d529189f6 (table test-table [id=36ce8e26297c46e58bb1204626de3d0c]) successfully deleted
I20260927 15:13:29.791257 14994 ts_tablet_manager.cc:1918] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:13:29.791465 14994 log.cc:1196] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/472da491194a428496c40d8d529189f6
I20260927 15:13:29.792052 14994 ts_tablet_manager.cc:1939] T 472da491194a428496c40d8d529189f6 P 8380ecc66ca14d8ba6164faaa97e3b4e: Deleting consensus metadata
I20260927 15:13:29.794582 14483 catalog_manager.cc:4841] TS 8380ecc66ca14d8ba6164faaa97e3b4e (127.10.33.67:33865): tablet 472da491194a428496c40d8d529189f6 (table test-table [id=36ce8e26297c46e58bb1204626de3d0c]) successfully deleted
I20260927 15:13:29.795306 14993 ts_tablet_manager.cc:1918] T 472da491194a428496c40d8d529189f6 P 1651e92a6d51470f9a346a053e2e8e02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:13:29.795498 14993 log.cc:1196] T 472da491194a428496c40d8d529189f6 P 1651e92a6d51470f9a346a053e2e8e02: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/472da491194a428496c40d8d529189f6
I20260927 15:13:29.795991 14993 ts_tablet_manager.cc:1939] T 472da491194a428496c40d8d529189f6 P 1651e92a6d51470f9a346a053e2e8e02: Deleting consensus metadata
Sep 27 15:13:29 dist-test-slave-f5h6 krb5kdc[14455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.65@KRBTEST.COM
I20260927 15:13:29.797896 14483 catalog_manager.cc:4841] TS 1651e92a6d51470f9a346a053e2e8e02 (127.10.33.65:38707): tablet 472da491194a428496c40d8d529189f6 (table test-table [id=36ce8e26297c46e58bb1204626de3d0c]) successfully deleted
I20260927 15:13:29.803364 14965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:29.771155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.10.33.65:38707)
0927 15:13:29.771324 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:29.771342 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:29.780096 (+ 8754us) server_negotiation.cc:407] Connection header received
0927 15:13:29.780606 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:29.780612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:29.780697 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:29.780769 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:29.782621 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.783801 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.791861 (+ 8060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.792653 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.792769 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:29.797526 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:29.797538 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:29.797552 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:29.797591 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:29.800501 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:29.801265 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:29.801273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:29.801275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:29.801415 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:29.801780 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:29.801784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:29.801786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:29.802236 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:29.802521 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:29.802818 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 15:13:29.803037 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:13:29.803943 14682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:29.927127 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 14560
I20260927 15:13:29.940976 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 14693
Sep 27 15:13:29 dist-test-slave-f5h6 krb5kdc[14455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.66@KRBTEST.COM
I20260927 15:13:29.959545 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:29.941837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35726 (local address 127.10.33.66:40163)
0927 15:13:29.942050 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:29.942056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:29.942078 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:13:29.942203 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:29.942209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:29.942288 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:29.942405 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:29.943551 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.944686 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.946031 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:29.946651 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:29.946785 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:29.952727 (+ 5942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:29.952745 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:29.952763 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:29.952814 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:29.956157 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:29.957145 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:29.957153 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:29.957156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:29.957300 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:29.958011 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:29.958015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:29.958017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:29.958444 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:29.958665 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:29.959036 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 15:13:29.959229 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 15:13:29.960265 14815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:30.056406 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 14693
I20260927 15:13:30.070453 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 14825
Sep 27 15:13:30 dist-test-slave-f5h6 krb5kdc[14455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.67@KRBTEST.COM
I20260927 15:13:30.088205 14969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:30.071326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38480 (local address 127.10.33.67:33865)
0927 15:13:30.071664 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:30.071670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:30.071697 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:13:30.071759 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:30.071765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:30.071843 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:30.071962 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:30.073337 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:30.074628 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:30.075949 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:30.076662 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:30.076802 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:30.081019 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:30.081036 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:30.081065 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:30.081118 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:30.085158 (+ 4040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:30.085946 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:30.085954 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:30.085956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:30.086144 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:30.086515 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:30.086519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:30.086522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:30.086880 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:30.087188 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:30.087459 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 15:13:30.087797 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 15:13:30.089114 14947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:30.212302 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 14825
I20260927 15:13:30.227723 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 14468
I20260927 15:13:30.228860 14529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:30.336794 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 14468
2026-09-27T15:13:30Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4406 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T15:13:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:13:30Z Disabled control of system clock
I20260927 15:13:30.399282 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:38137
--webserver_interface=127.10.33.126
--webserver_port=0
--builtin_ntp_servers=127.10.33.84:44657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:38137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 15:13:30.727347 15003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:30.727679 15003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:30.727787 15003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:30.735244 15003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:13:30.735371 15003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:30.735442 15003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:30.735524 15003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:13:30.735576 15003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:13:30.745111 15003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:44657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:38137
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:38137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:30.746563 15003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:30.748412 15003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:30.760509 15009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:30.760601 15008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:30.760579 15011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:30.761289 15003 server_base.cc:1034] running on GCE node
I20260927 15:13:30.762193 15003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:30.763839 15003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:30.765062 15003 hybrid_clock.cc:648] HybridClock initialized: now 1790522010765018 us; error 43 us; skew 500 ppm
I20260927 15:13:30.765444 15003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:13:30.767735 15003 webserver.cc:460] Webserver started at http://127.10.33.126:46291/ using document root <none> and password file <none>
I20260927 15:13:30.768436 15003 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:30.768522 15003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:30.768826 15003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:30.771721 15003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "a290b615a4e44a88b44eb5652bdaea25"
format_stamp: "Formatted at 2026-09-27 15:13:30 on dist-test-slave-f5h6"
server_key: "27bd897607aa03f3509eb3adfe0848e5"
server_key_iv: "acc5bf7ed01d281d5457f75218895e07"
server_key_version: "encryptionkey@0"
I20260927 15:13:30.772516 15003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "a290b615a4e44a88b44eb5652bdaea25"
format_stamp: "Formatted at 2026-09-27 15:13:30 on dist-test-slave-f5h6"
server_key: "27bd897607aa03f3509eb3adfe0848e5"
server_key_iv: "acc5bf7ed01d281d5457f75218895e07"
server_key_version: "encryptionkey@0"
I20260927 15:13:30.778234 15003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 15:13:30.781738 15017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:30.783385 15003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:13:30.783674 15003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "a290b615a4e44a88b44eb5652bdaea25"
format_stamp: "Formatted at 2026-09-27 15:13:30 on dist-test-slave-f5h6"
server_key: "27bd897607aa03f3509eb3adfe0848e5"
server_key_iv: "acc5bf7ed01d281d5457f75218895e07"
server_key_version: "encryptionkey@0"
I20260927 15:13:30.783937 15003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:30.790153 15003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:30.790985 15003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:30.791512 15003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:30.809336 15003 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:38137
I20260927 15:13:30.809352 15068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:38137 every 8 connection(s)
I20260927 15:13:30.810775 15003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:13:30.814666 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:30.818701 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 15003
I20260927 15:13:30.818913 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:13:30.819680 10373 external_mini_cluster.cc:1428] Setting key 0d97a35c2d8029d97ab49987d42262cf
I20260927 15:13:30.826962 15069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25: Bootstrap starting.
I20260927 15:13:30.829162 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:30.822008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.10.33.126:38137)
0927 15:13:30.823018 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:30.823030 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:30.823076 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:13:30.825405 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:30.825434 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:30.825994 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:30.826403 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:30.827522 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:30.827529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:30.827543 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:13:30.827558 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:30.827616 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:13:30.827631 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:30.827827 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:30.827978 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 15:13:30.828227 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":372,"threads_started":1}
I20260927 15:13:30.830566 15069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:30.831593 15069 log.cc:824] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:30.834530 15069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25: No bootstrap required, opened a new log
I20260927 15:13:30.842434 15069 raft_consensus.cc:348] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a290b615a4e44a88b44eb5652bdaea25" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 38137 } }
I20260927 15:13:30.842849 15069 raft_consensus.cc:374] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:30.842952 15069 raft_consensus.cc:729] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a290b615a4e44a88b44eb5652bdaea25, State: Initialized, Role: FOLLOWER
I20260927 15:13:30.843591 15069 consensus_queue.cc:260] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a290b615a4e44a88b44eb5652bdaea25" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 38137 } }
I20260927 15:13:30.843878 15069 raft_consensus.cc:388] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:13:30.844027 15069 raft_consensus.cc:482] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:13:30.844213 15069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:30.846490 15069 raft_consensus.cc:504] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a290b615a4e44a88b44eb5652bdaea25" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 38137 } }
I20260927 15:13:30.847046 15069 leader_election.cc:302] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a290b615a4e44a88b44eb5652bdaea25; no voters:
I20260927 15:13:30.847718 15069 leader_election.cc:288] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:13:30.847865 15074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:30.848512 15074 raft_consensus.cc:686] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25 [term 1 LEADER]: Becoming Leader. State: Replica: a290b615a4e44a88b44eb5652bdaea25, State: Running, Role: LEADER
I20260927 15:13:30.849033 15074 consensus_queue.cc:237] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a290b615a4e44a88b44eb5652bdaea25" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 38137 } }
I20260927 15:13:30.849727 15069 sys_catalog.cc:564] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:13:30.852459 15075 sys_catalog.cc:455] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a290b615a4e44a88b44eb5652bdaea25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a290b615a4e44a88b44eb5652bdaea25" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 38137 } } }
I20260927 15:13:30.852643 15076 sys_catalog.cc:455] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a290b615a4e44a88b44eb5652bdaea25. Latest consensus state: current_term: 1 leader_uuid: "a290b615a4e44a88b44eb5652bdaea25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a290b615a4e44a88b44eb5652bdaea25" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 38137 } } }
I20260927 15:13:30.852846 15075 sys_catalog.cc:458] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:30.852919 15076 sys_catalog.cc:458] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:30.854378 15083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:13:30.860540 15083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:13:30.876744 15083 catalog_manager.cc:1347] Generated new cluster ID: 9fab6be11f1e4df1ab6a839e6a69ac68
I20260927 15:13:30.876876 15083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:13:30.890992 15083 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:13:30.891539 15083 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:13:30.903777 15083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25: Generated new TSK 0
I20260927 15:13:30.904973 15083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:13:30.920248 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:38137
--builtin_ntp_servers=127.10.33.84:44657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 15:13:31.267055 15093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:31.267333 15093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:31.267449 15093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:31.274685 15093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:31.274791 15093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:31.274933 15093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:13:31.284420 15093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:44657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:38137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:31.285821 15093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:31.287705 15093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:31.301174 15098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:31.301141 15101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:31.301234 15099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:31.301781 15093 server_base.cc:1034] running on GCE node
I20260927 15:13:31.302512 15093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:31.303633 15093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:31.304884 15093 hybrid_clock.cc:648] HybridClock initialized: now 1790522011304793 us; error 73 us; skew 500 ppm
I20260927 15:13:31.305217 15093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:13:31.315217 15093 webserver.cc:460] Webserver started at http://127.10.33.65:45707/ using document root <none> and password file <none>
I20260927 15:13:31.316023 15093 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:31.316140 15093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:31.316473 15093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:31.319587 15093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "3c55bbf9a5c14aaa8be778b97c701fe2"
format_stamp: "Formatted at 2026-09-27 15:13:31 on dist-test-slave-f5h6"
server_key: "25399c0676ac96db7915d787669fd628"
server_key_iv: "06f0b461dcd29e827691ce5188be0f32"
server_key_version: "encryptionkey@0"
I20260927 15:13:31.320448 15093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "3c55bbf9a5c14aaa8be778b97c701fe2"
format_stamp: "Formatted at 2026-09-27 15:13:31 on dist-test-slave-f5h6"
server_key: "25399c0676ac96db7915d787669fd628"
server_key_iv: "06f0b461dcd29e827691ce5188be0f32"
server_key_version: "encryptionkey@0"
I20260927 15:13:31.326187 15093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 15:13:31.329963 15107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:31.331494 15093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:13:31.331674 15093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "3c55bbf9a5c14aaa8be778b97c701fe2"
format_stamp: "Formatted at 2026-09-27 15:13:31 on dist-test-slave-f5h6"
server_key: "25399c0676ac96db7915d787669fd628"
server_key_iv: "06f0b461dcd29e827691ce5188be0f32"
server_key_version: "encryptionkey@0"
I20260927 15:13:31.331935 15093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:31.337534 15093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:31.338404 15093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:31.338649 15093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:31.339576 15093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:31.340955 15093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:31.341045 15093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:31.341146 15093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:31.341209 15093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:31.372191 15093 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:43613
I20260927 15:13:31.372262 15219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:43613 every 8 connection(s)
I20260927 15:13:31.373533 15093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:13:31.382157 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:31.376195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:47281 (local address 127.10.33.126:38137)
0927 15:13:31.376605 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:31.376609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:31.377757 (+ 1148us) server_negotiation.cc:407] Connection header received
0927 15:13:31.379502 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:31.379509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:31.379571 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:31.379665 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:31.380991 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:31.380998 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:31.381002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:13:31.381085 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:31.381149 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:13:31.381168 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:31.381315 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:31.381679 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 15:13:31.381785 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":227,"threads_started":1}
I20260927 15:13:31.382330 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 15093
I20260927 15:13:31.382705 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:13:31.383258 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:31.376658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:38137 (local address 127.10.33.65:47281)
0927 15:13:31.377579 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:31.377614 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:31.379219 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:31.379910 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:31.379919 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:31.380029 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:31.380687 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:13:31.380689 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:31.380729 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:13:31.380820 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:31.381301 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:31.381312 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:31.381320 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 15:13:31.381754 (+ 434us) client_negotiation.cc:240] Negotiation successful
0927 15:13:31.382188 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":295,"threads_started":1}
I20260927 15:13:31.383343 10373 external_mini_cluster.cc:1428] Setting key 0f13b62c5c86bcf1533ffdad4cb5fc02
I20260927 15:13:31.385097 15220 heartbeater.cc:347] Connected to a master server at 127.10.33.126:38137
I20260927 15:13:31.385463 15220 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:31.386409 15220 heartbeater.cc:511] Master 127.10.33.126:38137 requested a full tablet report, sending...
I20260927 15:13:31.387266 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:38137
--builtin_ntp_servers=127.10.33.84:44657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 15:13:31.389281 15034 ts_manager.cc:194] Registered new tserver with Master: 3c55bbf9a5c14aaa8be778b97c701fe2 (127.10.33.65:43613)
W20260927 15:13:31.717288 15224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:31.717588 15224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:31.717677 15224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:31.725095 15224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:31.725221 15224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:31.725360 15224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:13:31.734762 15224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:44657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:38137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:31.736150 15224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:31.738013 15224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:31.751673 15229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:31.751627 15230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:31.751627 15232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:31.752295 15224 server_base.cc:1034] running on GCE node
I20260927 15:13:31.752993 15224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:31.754034 15224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:31.755249 15224 hybrid_clock.cc:648] HybridClock initialized: now 1790522011755201 us; error 40 us; skew 500 ppm
I20260927 15:13:31.755579 15224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:13:31.757900 15224 webserver.cc:460] Webserver started at http://127.10.33.66:38557/ using document root <none> and password file <none>
I20260927 15:13:31.758654 15224 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:31.758759 15224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:31.759078 15224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:31.762045 15224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "1500e6fc39c24290a5053ffe6dd9326b"
format_stamp: "Formatted at 2026-09-27 15:13:31 on dist-test-slave-f5h6"
server_key: "c90a8e532233d31fa8c5fc6e63bdea1d"
server_key_iv: "3baddf856282a450361a3bb3ee76566c"
server_key_version: "encryptionkey@0"
I20260927 15:13:31.762802 15224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "1500e6fc39c24290a5053ffe6dd9326b"
format_stamp: "Formatted at 2026-09-27 15:13:31 on dist-test-slave-f5h6"
server_key: "c90a8e532233d31fa8c5fc6e63bdea1d"
server_key_iv: "3baddf856282a450361a3bb3ee76566c"
server_key_version: "encryptionkey@0"
I20260927 15:13:31.767882 15224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 15:13:31.771232 15238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:31.772508 15224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 15:13:31.772653 15224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "1500e6fc39c24290a5053ffe6dd9326b"
format_stamp: "Formatted at 2026-09-27 15:13:31 on dist-test-slave-f5h6"
server_key: "c90a8e532233d31fa8c5fc6e63bdea1d"
server_key_iv: "3baddf856282a450361a3bb3ee76566c"
server_key_version: "encryptionkey@0"
I20260927 15:13:31.772842 15224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:31.778932 15224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:31.779827 15224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:31.780081 15224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:31.781009 15224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:31.782466 15224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:31.782559 15224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:31.782670 15224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:31.782737 15224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:31.814998 15224 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:41177
I20260927 15:13:31.815021 15350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:41177 every 8 connection(s)
I20260927 15:13:31.816294 15224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:13:31.817202 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 15224
I20260927 15:13:31.817345 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:13:31.817917 10373 external_mini_cluster.cc:1428] Setting key e320a4790819f93582efd6444997c037
I20260927 15:13:31.820995 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:38137
--builtin_ntp_servers=127.10.33.84:44657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 15:13:31.828872 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:31.820566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:56603 (local address 127.10.33.126:38137)
0927 15:13:31.820737 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:31.820745 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:31.822922 (+ 2177us) server_negotiation.cc:407] Connection header received
0927 15:13:31.825786 (+ 2864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:31.825794 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:31.825922 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:31.825998 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:31.828030 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:31.828038 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:31.828043 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:13:31.828066 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:31.828109 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:13:31.828126 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:31.828221 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:31.828539 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 15:13:31.828609 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 15:13:31.830968 15353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:31.819918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:38137 (local address 127.10.33.66:56603)
0927 15:13:31.822750 (+ 2832us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:31.822798 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:31.825389 (+ 2591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:31.826614 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:31.826627 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:31.826786 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:31.827600 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:13:31.827605 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:31.827670 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:13:31.827800 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:31.828309 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:31.828319 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:31.828327 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 15:13:31.828641 (+ 314us) client_negotiation.cc:240] Negotiation successful
0927 15:13:31.830068 (+ 1427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2244,"thread_start_us":2007,"threads_started":1}
I20260927 15:13:31.832422 15351 heartbeater.cc:347] Connected to a master server at 127.10.33.126:38137
I20260927 15:13:31.832751 15351 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:31.833412 15351 heartbeater.cc:511] Master 127.10.33.126:38137 requested a full tablet report, sending...
I20260927 15:13:31.835062 15034 ts_manager.cc:194] Registered new tserver with Master: 1500e6fc39c24290a5053ffe6dd9326b (127.10.33.66:41177)
W20260927 15:13:32.159772 15354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:32.160060 15354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:32.160147 15354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:32.167467 15354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:32.167578 15354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:32.167686 15354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:13:32.176666 15354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:44657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:38137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:32.178001 15354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:32.179657 15354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:32.192174 15359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:32.192207 15360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:32.192256 15362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:32.193235 15354 server_base.cc:1034] running on GCE node
I20260927 15:13:32.193969 15354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:32.194998 15354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:32.196218 15354 hybrid_clock.cc:648] HybridClock initialized: now 1790522012196158 us; error 49 us; skew 500 ppm
I20260927 15:13:32.196565 15354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:13:32.199121 15354 webserver.cc:460] Webserver started at http://127.10.33.67:34845/ using document root <none> and password file <none>
I20260927 15:13:32.199808 15354 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:32.199910 15354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:32.200237 15354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:32.203125 15354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "1a6fcb20d4bc482fafcdf485f298f366"
format_stamp: "Formatted at 2026-09-27 15:13:32 on dist-test-slave-f5h6"
server_key: "808cb3e5f7a20b46396b72483c74b31a"
server_key_iv: "6453b735fc56de5b660f8fc2a4318296"
server_key_version: "encryptionkey@0"
I20260927 15:13:32.203871 15354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "1a6fcb20d4bc482fafcdf485f298f366"
format_stamp: "Formatted at 2026-09-27 15:13:32 on dist-test-slave-f5h6"
server_key: "808cb3e5f7a20b46396b72483c74b31a"
server_key_iv: "6453b735fc56de5b660f8fc2a4318296"
server_key_version: "encryptionkey@0"
I20260927 15:13:32.209115 15354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260927 15:13:32.212816 15368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:32.214214 15354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:13:32.214372 15354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "1a6fcb20d4bc482fafcdf485f298f366"
format_stamp: "Formatted at 2026-09-27 15:13:32 on dist-test-slave-f5h6"
server_key: "808cb3e5f7a20b46396b72483c74b31a"
server_key_iv: "6453b735fc56de5b660f8fc2a4318296"
server_key_version: "encryptionkey@0"
I20260927 15:13:32.214587 15354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:32.220512 15354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:32.221352 15354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:32.221601 15354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:32.222579 15354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:32.223928 15354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:32.224014 15354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:32.224095 15354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:32.224148 15354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:32.253521 15354 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:39499
I20260927 15:13:32.253571 15480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:39499 every 8 connection(s)
I20260927 15:13:32.254796 15354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:13:32.255131 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 15354
I20260927 15:13:32.255299 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:13:32.255823 10373 external_mini_cluster.cc:1428] Setting key aaa699cfdd88216c13415862165e9930
I20260927 15:13:32.263015 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.257460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:33587 (local address 127.10.33.126:38137)
0927 15:13:32.257660 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:32.257666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:32.259279 (+ 1613us) server_negotiation.cc:407] Connection header received
0927 15:13:32.260897 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:32.260903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:32.260985 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:32.261072 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:32.262330 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:32.262337 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:32.262342 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:13:32.262360 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:32.262391 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:13:32.262402 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:32.262501 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:32.262671 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 15:13:32.262733 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:13:32.263893 15483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.258126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:38137 (local address 127.10.33.67:33587)
0927 15:13:32.259146 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:32.259187 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:32.260650 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:32.261294 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:32.261302 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:32.261401 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:32.262048 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:13:32.262050 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:32.262086 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:13:32.262170 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:32.262485 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:32.262495 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:32.262502 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:13:32.262739 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 15:13:32.262989 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":310,"threads_started":1}
I20260927 15:13:32.265197 15481 heartbeater.cc:347] Connected to a master server at 127.10.33.126:38137
I20260927 15:13:32.265491 15481 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:32.266119 15481 heartbeater.cc:511] Master 127.10.33.126:38137 requested a full tablet report, sending...
I20260927 15:13:32.267454 15034 ts_manager.cc:194] Registered new tserver with Master: 1a6fcb20d4bc482fafcdf485f298f366 (127.10.33.67:39499)
I20260927 15:13:32.271100 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:13:32.277534 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.275287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56346 (local address 127.10.33.126:38137)
0927 15:13:32.275432 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:32.275437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:32.275887 (+ 450us) server_negotiation.cc:407] Connection header received
0927 15:13:32.276086 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:32.276089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:32.276158 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:32.276229 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:32.277022 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:32.277026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:32.277029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:13:32.277043 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:32.277076 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:13:32.277092 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:32.277189 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:32.277261 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 15:13:32.277304 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:13:32.281272 15034 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:13:32.288298 15034 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:13:32.310192 15493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.300441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:39499 (local address 127.0.0.1:51798)
0927 15:13:32.301200 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:32.301243 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:32.301400 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:32.307886 (+ 6486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:32.307891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:32.307912 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:32.308854 (+ 942us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:13:32.308858 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:32.308893 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:13:32.308978 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:32.309474 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:32.309484 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:32.309493 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 15:13:32.309640 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 15:13:32.309727 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":997,"threads_started":1}
I20260927 15:13:32.310662 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.300694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51798 (local address 127.10.33.67:39499)
0927 15:13:32.307378 (+ 6684us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:32.307386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:32.307423 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:13:32.307493 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:32.307500 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:32.307705 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:32.308191 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:32.309232 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:32.309241 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:32.309256 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:13:32.309272 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:32.309325 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:13:32.309345 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:32.310125 (+ 780us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:32.310195 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 15:13:32.310295 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6532,"thread_start_us":6354,"threads_started":1}
I20260927 15:13:32.311151 15497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.300684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:43613 (local address 127.0.0.1:40192)
0927 15:13:32.302909 (+ 2225us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:32.302931 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:32.303144 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:32.307758 (+ 4614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:32.307768 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:32.307868 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:32.309905 (+ 2037us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:13:32.309909 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:32.309928 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:13:32.309976 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:32.310442 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:32.310448 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:32.310454 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:13:32.310584 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 15:13:32.310637 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2071,"thread_start_us":6711,"threads_started":1}
I20260927 15:13:32.311228 15494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.302331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:41177 (local address 127.0.0.1:43016)
0927 15:13:32.307689 (+ 5358us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:32.307708 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:32.307817 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:32.309243 (+ 1426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:32.309247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:32.309270 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:32.309917 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:13:32.309920 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:32.309933 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:13:32.309976 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:32.310525 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:32.310530 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:32.310535 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:13:32.310668 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 15:13:32.310711 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5201,"thread_start_us":3574,"threads_started":1}
I20260927 15:13:32.312003 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.302915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.10.33.66:41177)
0927 15:13:32.306566 (+ 3651us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:32.306579 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:32.307930 (+ 1351us) server_negotiation.cc:407] Connection header received
0927 15:13:32.308003 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:32.308010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:32.308197 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:32.308328 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:32.310170 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:32.310180 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:32.310195 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:13:32.310225 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:32.310286 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:13:32.310308 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:32.310475 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:32.311399 (+ 924us) server_negotiation.cc:299] Negotiation successful
0927 15:13:32.311506 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3442,"thread_start_us":3275,"threads_started":1}
I20260927 15:13:32.312062 15496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.301077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40192 (local address 127.10.33.65:43613)
0927 15:13:32.307033 (+ 5956us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:32.307043 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:32.307085 (+ 42us) server_negotiation.cc:407] Connection header received
0927 15:13:32.307175 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:32.307185 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:32.307478 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:32.307711 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:32.310171 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:32.310180 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:32.310195 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:13:32.310225 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:32.310286 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:13:32.310320 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:32.311566 (+ 1246us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:32.311659 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 15:13:32.311748 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5783,"thread_start_us":5641,"threads_started":1}
I20260927 15:13:32.314632 15416 tablet_service.cc:1467] Processing CreateTablet for tablet d4fc9136a656430e8fa066f91bd7aefe (DEFAULT_TABLE table=test-table [id=7a3f8c964c7545e695dccbe4b07f6949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:32.315840 15416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4fc9136a656430e8fa066f91bd7aefe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:32.315619 15155 tablet_service.cc:1467] Processing CreateTablet for tablet d4fc9136a656430e8fa066f91bd7aefe (DEFAULT_TABLE table=test-table [id=7a3f8c964c7545e695dccbe4b07f6949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:32.315619 15286 tablet_service.cc:1467] Processing CreateTablet for tablet d4fc9136a656430e8fa066f91bd7aefe (DEFAULT_TABLE table=test-table [id=7a3f8c964c7545e695dccbe4b07f6949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:32.316970 15286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4fc9136a656430e8fa066f91bd7aefe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:32.316972 15155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4fc9136a656430e8fa066f91bd7aefe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:32.327667 15499 tablet_bootstrap.cc:492] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366: Bootstrap starting.
I20260927 15:13:32.331408 15499 tablet_bootstrap.cc:654] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:32.332789 15500 tablet_bootstrap.cc:492] T d4fc9136a656430e8fa066f91bd7aefe P 1500e6fc39c24290a5053ffe6dd9326b: Bootstrap starting.
I20260927 15:13:32.334470 15499 log.cc:824] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:32.334619 15501 tablet_bootstrap.cc:492] T d4fc9136a656430e8fa066f91bd7aefe P 3c55bbf9a5c14aaa8be778b97c701fe2: Bootstrap starting.
I20260927 15:13:32.337039 15500 tablet_bootstrap.cc:654] T d4fc9136a656430e8fa066f91bd7aefe P 1500e6fc39c24290a5053ffe6dd9326b: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:32.337591 15501 tablet_bootstrap.cc:654] T d4fc9136a656430e8fa066f91bd7aefe P 3c55bbf9a5c14aaa8be778b97c701fe2: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:32.338521 15501 log.cc:824] T d4fc9136a656430e8fa066f91bd7aefe P 3c55bbf9a5c14aaa8be778b97c701fe2: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:32.338452 15500 log.cc:824] T d4fc9136a656430e8fa066f91bd7aefe P 1500e6fc39c24290a5053ffe6dd9326b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:32.341102 15501 tablet_bootstrap.cc:492] T d4fc9136a656430e8fa066f91bd7aefe P 3c55bbf9a5c14aaa8be778b97c701fe2: No bootstrap required, opened a new log
I20260927 15:13:32.341349 15501 ts_tablet_manager.cc:1397] T d4fc9136a656430e8fa066f91bd7aefe P 3c55bbf9a5c14aaa8be778b97c701fe2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 15:13:32.344069 15499 tablet_bootstrap.cc:492] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366: No bootstrap required, opened a new log
I20260927 15:13:32.344339 15499 ts_tablet_manager.cc:1397] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20260927 15:13:32.344710 15500 tablet_bootstrap.cc:492] T d4fc9136a656430e8fa066f91bd7aefe P 1500e6fc39c24290a5053ffe6dd9326b: No bootstrap required, opened a new log
I20260927 15:13:32.344975 15500 ts_tablet_manager.cc:1397] T d4fc9136a656430e8fa066f91bd7aefe P 1500e6fc39c24290a5053ffe6dd9326b: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260927 15:13:32.351684 15499 raft_consensus.cc:348] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6fcb20d4bc482fafcdf485f298f366" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 39499 } } peers { permanent_uuid: "3c55bbf9a5c14aaa8be778b97c701fe2" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 43613 } } peers { permanent_uuid: "1500e6fc39c24290a5053ffe6dd9326b" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 41177 } }
I20260927 15:13:32.351609 15501 raft_consensus.cc:348] T d4fc9136a656430e8fa066f91bd7aefe P 3c55bbf9a5c14aaa8be778b97c701fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6fcb20d4bc482fafcdf485f298f366" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 39499 } } peers { permanent_uuid: "3c55bbf9a5c14aaa8be778b97c701fe2" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 43613 } } peers { permanent_uuid: "1500e6fc39c24290a5053ffe6dd9326b" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 41177 } }
I20260927 15:13:32.352063 15499 raft_consensus.cc:374] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:32.352097 15501 raft_consensus.cc:374] T d4fc9136a656430e8fa066f91bd7aefe P 3c55bbf9a5c14aaa8be778b97c701fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:32.352159 15499 raft_consensus.cc:729] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a6fcb20d4bc482fafcdf485f298f366, State: Initialized, Role: FOLLOWER
I20260927 15:13:32.352186 15501 raft_consensus.cc:729] T d4fc9136a656430e8fa066f91bd7aefe P 3c55bbf9a5c14aaa8be778b97c701fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c55bbf9a5c14aaa8be778b97c701fe2, State: Initialized, Role: FOLLOWER
I20260927 15:13:32.352931 15501 consensus_queue.cc:260] T d4fc9136a656430e8fa066f91bd7aefe P 3c55bbf9a5c14aaa8be778b97c701fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6fcb20d4bc482fafcdf485f298f366" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 39499 } } peers { permanent_uuid: "3c55bbf9a5c14aaa8be778b97c701fe2" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 43613 } } peers { permanent_uuid: "1500e6fc39c24290a5053ffe6dd9326b" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 41177 } }
I20260927 15:13:32.352924 15499 consensus_queue.cc:260] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6fcb20d4bc482fafcdf485f298f366" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 39499 } } peers { permanent_uuid: "3c55bbf9a5c14aaa8be778b97c701fe2" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 43613 } } peers { permanent_uuid: "1500e6fc39c24290a5053ffe6dd9326b" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 41177 } }
I20260927 15:13:32.354929 15481 heartbeater.cc:503] Master 127.10.33.126:38137 was elected leader, sending a full tablet report...
I20260927 15:13:32.355643 15499 ts_tablet_manager.cc:1428] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260927 15:13:32.355481 15500 raft_consensus.cc:348] T d4fc9136a656430e8fa066f91bd7aefe P 1500e6fc39c24290a5053ffe6dd9326b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6fcb20d4bc482fafcdf485f298f366" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 39499 } } peers { permanent_uuid: "3c55bbf9a5c14aaa8be778b97c701fe2" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 43613 } } peers { permanent_uuid: "1500e6fc39c24290a5053ffe6dd9326b" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 41177 } }
I20260927 15:13:32.355973 15500 raft_consensus.cc:374] T d4fc9136a656430e8fa066f91bd7aefe P 1500e6fc39c24290a5053ffe6dd9326b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:32.356263 15500 raft_consensus.cc:729] T d4fc9136a656430e8fa066f91bd7aefe P 1500e6fc39c24290a5053ffe6dd9326b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1500e6fc39c24290a5053ffe6dd9326b, State: Initialized, Role: FOLLOWER
I20260927 15:13:32.357002 15500 consensus_queue.cc:260] T d4fc9136a656430e8fa066f91bd7aefe P 1500e6fc39c24290a5053ffe6dd9326b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6fcb20d4bc482fafcdf485f298f366" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 39499 } } peers { permanent_uuid: "3c55bbf9a5c14aaa8be778b97c701fe2" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 43613 } } peers { permanent_uuid: "1500e6fc39c24290a5053ffe6dd9326b" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 41177 } }
I20260927 15:13:32.357273 15220 heartbeater.cc:503] Master 127.10.33.126:38137 was elected leader, sending a full tablet report...
I20260927 15:13:32.358052 15501 ts_tablet_manager.cc:1428] T d4fc9136a656430e8fa066f91bd7aefe P 3c55bbf9a5c14aaa8be778b97c701fe2: Time spent starting tablet: real 0.017s user 0.009s sys 0.009s
I20260927 15:13:32.358904 15351 heartbeater.cc:503] Master 127.10.33.126:38137 was elected leader, sending a full tablet report...
I20260927 15:13:32.359735 15500 ts_tablet_manager.cc:1428] T d4fc9136a656430e8fa066f91bd7aefe P 1500e6fc39c24290a5053ffe6dd9326b: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
W20260927 15:13:32.375802 15221 tablet.cc:2369] T d4fc9136a656430e8fa066f91bd7aefe P 3c55bbf9a5c14aaa8be778b97c701fe2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:13:32.470551 15506 raft_consensus.cc:482] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:13:32.470873 15506 raft_consensus.cc:504] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6fcb20d4bc482fafcdf485f298f366" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 39499 } } peers { permanent_uuid: "3c55bbf9a5c14aaa8be778b97c701fe2" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 43613 } } peers { permanent_uuid: "1500e6fc39c24290a5053ffe6dd9326b" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 41177 } }
I20260927 15:13:32.472676 15506 leader_election.cc:288] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c55bbf9a5c14aaa8be778b97c701fe2 (127.10.33.65:43613), 1500e6fc39c24290a5053ffe6dd9326b (127.10.33.66:41177)
I20260927 15:13:32.475703 15496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.473269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:57243 (local address 127.10.33.65:43613)
0927 15:13:32.473485 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:32.473491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:32.473737 (+ 246us) server_negotiation.cc:407] Connection header received
0927 15:13:32.473809 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:32.473814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:32.473901 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:32.473997 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:32.475093 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:32.475098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:32.475101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:13:32.475116 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:32.475145 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:13:32.475156 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:32.475254 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:32.475344 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 15:13:32.475437 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 15:13:32.475682 15483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.472945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:43613 (local address 127.10.33.67:57243)
0927 15:13:32.473482 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:32.473501 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:32.473652 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:32.474039 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:32.474048 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:32.474073 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:32.474848 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:13:32.474850 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:32.474864 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:13:32.474942 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:32.475243 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:32.475247 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:32.475252 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:13:32.475349 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 15:13:32.475401 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421}
I20260927 15:13:32.475857 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.473520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:50613 (local address 127.10.33.66:41177)
0927 15:13:32.473692 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:32.473698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:32.474144 (+ 446us) server_negotiation.cc:407] Connection header received
0927 15:13:32.474275 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:32.474280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:32.474337 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:32.474431 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:32.475328 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:32.475334 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:32.475338 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:13:32.475355 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:32.475384 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:13:32.475395 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:32.475503 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:32.475561 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 15:13:32.475617 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:13:32.476258 15508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.473393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:41177 (local address 127.10.33.67:50613)
0927 15:13:32.474060 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:32.474074 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:32.474172 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:32.474520 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:32.474524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:32.474548 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:32.475137 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:13:32.475139 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:32.475152 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:13:32.475187 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:32.475478 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:32.475482 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:32.475487 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:13:32.475585 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 15:13:32.475629 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":328,"threads_started":1}
I20260927 15:13:32.476816 15175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fc9136a656430e8fa066f91bd7aefe" candidate_uuid: "1a6fcb20d4bc482fafcdf485f298f366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c55bbf9a5c14aaa8be778b97c701fe2" is_pre_election: true
I20260927 15:13:32.477073 15306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fc9136a656430e8fa066f91bd7aefe" candidate_uuid: "1a6fcb20d4bc482fafcdf485f298f366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1500e6fc39c24290a5053ffe6dd9326b" is_pre_election: true
I20260927 15:13:32.477324 15175 raft_consensus.cc:2454] T d4fc9136a656430e8fa066f91bd7aefe P 3c55bbf9a5c14aaa8be778b97c701fe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a6fcb20d4bc482fafcdf485f298f366 in term 0.
I20260927 15:13:32.477571 15306 raft_consensus.cc:2454] T d4fc9136a656430e8fa066f91bd7aefe P 1500e6fc39c24290a5053ffe6dd9326b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a6fcb20d4bc482fafcdf485f298f366 in term 0.
I20260927 15:13:32.478153 15370 leader_election.cc:302] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1500e6fc39c24290a5053ffe6dd9326b, 1a6fcb20d4bc482fafcdf485f298f366, 3c55bbf9a5c14aaa8be778b97c701fe2; no voters:
I20260927 15:13:32.478590 15506 raft_consensus.cc:2781] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:13:32.478719 15506 raft_consensus.cc:482] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:13:32.478816 15506 raft_consensus.cc:3037] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:32.481042 15506 raft_consensus.cc:504] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6fcb20d4bc482fafcdf485f298f366" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 39499 } } peers { permanent_uuid: "3c55bbf9a5c14aaa8be778b97c701fe2" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 43613 } } peers { permanent_uuid: "1500e6fc39c24290a5053ffe6dd9326b" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 41177 } }
I20260927 15:13:32.481679 15506 leader_election.cc:288] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366 [CANDIDATE]: Term 1 election: Requested vote from peers 3c55bbf9a5c14aaa8be778b97c701fe2 (127.10.33.65:43613), 1500e6fc39c24290a5053ffe6dd9326b (127.10.33.66:41177)
I20260927 15:13:32.482300 15306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fc9136a656430e8fa066f91bd7aefe" candidate_uuid: "1a6fcb20d4bc482fafcdf485f298f366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1500e6fc39c24290a5053ffe6dd9326b"
I20260927 15:13:32.482343 15175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fc9136a656430e8fa066f91bd7aefe" candidate_uuid: "1a6fcb20d4bc482fafcdf485f298f366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c55bbf9a5c14aaa8be778b97c701fe2"
I20260927 15:13:32.482503 15306 raft_consensus.cc:3037] T d4fc9136a656430e8fa066f91bd7aefe P 1500e6fc39c24290a5053ffe6dd9326b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:32.482539 15175 raft_consensus.cc:3037] T d4fc9136a656430e8fa066f91bd7aefe P 3c55bbf9a5c14aaa8be778b97c701fe2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:32.485088 15175 raft_consensus.cc:2454] T d4fc9136a656430e8fa066f91bd7aefe P 3c55bbf9a5c14aaa8be778b97c701fe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a6fcb20d4bc482fafcdf485f298f366 in term 1.
I20260927 15:13:32.485351 15306 raft_consensus.cc:2454] T d4fc9136a656430e8fa066f91bd7aefe P 1500e6fc39c24290a5053ffe6dd9326b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a6fcb20d4bc482fafcdf485f298f366 in term 1.
I20260927 15:13:32.485585 15370 leader_election.cc:302] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a6fcb20d4bc482fafcdf485f298f366, 3c55bbf9a5c14aaa8be778b97c701fe2; no voters:
I20260927 15:13:32.485898 15506 raft_consensus.cc:2781] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:32.486325 15506 raft_consensus.cc:686] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366 [term 1 LEADER]: Becoming Leader. State: Replica: 1a6fcb20d4bc482fafcdf485f298f366, State: Running, Role: LEADER
I20260927 15:13:32.486769 15506 consensus_queue.cc:237] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6fcb20d4bc482fafcdf485f298f366" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 39499 } } peers { permanent_uuid: "3c55bbf9a5c14aaa8be778b97c701fe2" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 43613 } } peers { permanent_uuid: "1500e6fc39c24290a5053ffe6dd9326b" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 41177 } }
I20260927 15:13:32.492008 15033 catalog_manager.cc:5495] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a6fcb20d4bc482fafcdf485f298f366 (127.10.33.67). New cstate: current_term: 1 leader_uuid: "1a6fcb20d4bc482fafcdf485f298f366" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6fcb20d4bc482fafcdf485f298f366" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 39499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c55bbf9a5c14aaa8be778b97c701fe2" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 43613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1500e6fc39c24290a5053ffe6dd9326b" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 41177 } health_report { overall_health: UNKNOWN } } }
W20260927 15:13:32.506532 15482 tablet.cc:2369] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:13:32.550614 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.548451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51812 (local address 127.10.33.67:39499)
0927 15:13:32.548629 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:32.548635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:32.548695 (+ 60us) server_negotiation.cc:407] Connection header received
0927 15:13:32.548912 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:32.548916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:32.549029 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:32.549118 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:32.550013 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:32.550019 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:32.550023 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:13:32.550044 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:32.550081 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:13:32.550095 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:32.550208 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:32.550308 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 15:13:32.550369 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:13:32.559000 15306 raft_consensus.cc:1261] T d4fc9136a656430e8fa066f91bd7aefe P 1500e6fc39c24290a5053ffe6dd9326b [term 1 FOLLOWER]: Refusing update from remote peer 1a6fcb20d4bc482fafcdf485f298f366: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:32.559098 15175 raft_consensus.cc:1261] T d4fc9136a656430e8fa066f91bd7aefe P 3c55bbf9a5c14aaa8be778b97c701fe2 [term 1 FOLLOWER]: Refusing update from remote peer 1a6fcb20d4bc482fafcdf485f298f366: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:32.560007 15509 consensus_queue.cc:1035] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c55bbf9a5c14aaa8be778b97c701fe2" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 43613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:32.560351 15506 consensus_queue.cc:1035] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1500e6fc39c24290a5053ffe6dd9326b" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 41177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:32.585704 15033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:32.586053 15033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:32.590252 15033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a290b615a4e44a88b44eb5652bdaea25: Sending DeleteTablet for 3 replicas of tablet d4fc9136a656430e8fa066f91bd7aefe
I20260927 15:13:32.591645 15155 tablet_service.cc:1514] Processing DeleteTablet for tablet d4fc9136a656430e8fa066f91bd7aefe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:32 UTC) from {username='slave'} at 127.0.0.1:40192
I20260927 15:13:32.591651 15416 tablet_service.cc:1514] Processing DeleteTablet for tablet d4fc9136a656430e8fa066f91bd7aefe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:32 UTC) from {username='slave'} at 127.0.0.1:51798
I20260927 15:13:32.592027 15286 tablet_service.cc:1514] Processing DeleteTablet for tablet d4fc9136a656430e8fa066f91bd7aefe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:32 UTC) from {username='slave'} at 127.0.0.1:43016
I20260927 15:13:32.594969 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 15093
I20260927 15:13:32.602049 15524 tablet_replica.cc:330] T d4fc9136a656430e8fa066f91bd7aefe P 1500e6fc39c24290a5053ffe6dd9326b: stopping tablet replica
I20260927 15:13:32.602552 15524 raft_consensus.cc:2229] T d4fc9136a656430e8fa066f91bd7aefe P 1500e6fc39c24290a5053ffe6dd9326b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:32.603109 15524 raft_consensus.cc:2258] T d4fc9136a656430e8fa066f91bd7aefe P 1500e6fc39c24290a5053ffe6dd9326b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:32.605708 15524 ts_tablet_manager.cc:1905] T d4fc9136a656430e8fa066f91bd7aefe P 1500e6fc39c24290a5053ffe6dd9326b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:32.611193 15496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.596408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40194 (local address 127.10.33.65:43613)
0927 15:13:32.596608 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:32.596615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:32.602335 (+ 5720us) server_negotiation.cc:407] Connection header received
0927 15:13:32.602623 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:32.602630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:32.602748 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:32.602826 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:32.610379 (+ 7553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:32.610386 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:32.610391 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:13:32.610416 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:32.610458 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:13:32.610473 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:32.610577 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:32.610808 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 15:13:32.610876 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:13:32.612290 15215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:32.612536 15524 ts_tablet_manager.cc:1918] T d4fc9136a656430e8fa066f91bd7aefe P 1500e6fc39c24290a5053ffe6dd9326b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:13:32.612701 15524 log.cc:1196] T d4fc9136a656430e8fa066f91bd7aefe P 1500e6fc39c24290a5053ffe6dd9326b: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/d4fc9136a656430e8fa066f91bd7aefe
I20260927 15:13:32.613227 15524 ts_tablet_manager.cc:1939] T d4fc9136a656430e8fa066f91bd7aefe P 1500e6fc39c24290a5053ffe6dd9326b: Deleting consensus metadata
I20260927 15:13:32.614449 15522 tablet_replica.cc:330] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366: stopping tablet replica
I20260927 15:13:32.614945 15522 raft_consensus.cc:2229] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:13:32.615602 15522 raft_consensus.cc:2258] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:32.617308 15021 catalog_manager.cc:4841] TS 1500e6fc39c24290a5053ffe6dd9326b (127.10.33.66:41177): tablet d4fc9136a656430e8fa066f91bd7aefe (table test-table [id=7a3f8c964c7545e695dccbe4b07f6949]) successfully deleted
I20260927 15:13:32.619269 15522 ts_tablet_manager.cc:1905] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:32.625173 15522 ts_tablet_manager.cc:1918] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:13:32.625340 15522 log.cc:1196] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/d4fc9136a656430e8fa066f91bd7aefe
I20260927 15:13:32.625905 15522 ts_tablet_manager.cc:1939] T d4fc9136a656430e8fa066f91bd7aefe P 1a6fcb20d4bc482fafcdf485f298f366: Deleting consensus metadata
I20260927 15:13:32.627810 15021 catalog_manager.cc:4841] TS 1a6fcb20d4bc482fafcdf485f298f366 (127.10.33.67:39499): tablet d4fc9136a656430e8fa066f91bd7aefe (table test-table [id=7a3f8c964c7545e695dccbe4b07f6949]) successfully deleted
I20260927 15:13:32.711474 15523 tablet_replica.cc:330] T d4fc9136a656430e8fa066f91bd7aefe P 3c55bbf9a5c14aaa8be778b97c701fe2: stopping tablet replica
I20260927 15:13:32.711527 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 15093
W20260927 15:13:32.731580 15019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.33.65:43613 (error 108)
I20260927 15:13:32.731798 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 15224
I20260927 15:13:32.733731 15494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.733231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:43613 (local address 127.0.0.1:40206)
0927 15:13:32.733428 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:32.733559 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:43613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 15:13:32.734017 15019 catalog_manager.cc:4584] TS 3c55bbf9a5c14aaa8be778b97c701fe2 (127.10.33.65:43613): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4fc9136a656430e8fa066f91bd7aefe: Network error: Client connection negotiation failed: client connection to 127.10.33.65:43613: connect: Connection refused (error 111)
I20260927 15:13:32.734938 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.732901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43022 (local address 127.10.33.66:41177)
0927 15:13:32.733090 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:32.733095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:32.733114 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:13:32.733176 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:32.733181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:32.733288 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:32.733382 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:32.734323 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:32.734329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:32.734334 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:13:32.734351 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:32.734382 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:13:32.734394 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:32.734510 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:32.734621 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 15:13:32.734683 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:13:32.735765 15346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:32.787731 15494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.787169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:43613 (local address 127.0.0.1:40216)
0927 15:13:32.787419 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:32.787531 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:43613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 15:13:32.788935 15494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.788530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:43613 (local address 127.0.0.1:40226)
0927 15:13:32.788673 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:32.788789 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:43613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 15:13:32.830667 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 15224
I20260927 15:13:32.844017 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 15354
I20260927 15:13:32.847043 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.844881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51828 (local address 127.10.33.67:39499)
0927 15:13:32.845102 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:32.845109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:32.845132 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:13:32.845224 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:32.845230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:32.845291 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:32.845413 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:32.846385 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:32.846391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:32.846395 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:13:32.846414 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:32.846449 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:13:32.846464 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:32.846610 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:32.846675 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 15:13:32.846731 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:13:32.847790 15476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:32.849383 15494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.848959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:43613 (local address 127.0.0.1:40230)
0927 15:13:32.849183 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:32.849266 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:43613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260927 15:13:32.850438 15494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.850082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:43613 (local address 127.0.0.1:40244)
0927 15:13:32.850222 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:32.850302 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:43613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 15:13:32.931085 15494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.930624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:43613 (local address 127.0.0.1:40258)
0927 15:13:32.930835 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:32.930927 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:43613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 15:13:32.932201 15494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:32.931835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:43613 (local address 127.0.0.1:40262)
0927 15:13:32.932007 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:32.932083 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:43613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 15:13:32.964790 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 15354
I20260927 15:13:32.979919 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 15003
I20260927 15:13:32.981153 15064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:33.088187 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 15003
2026-09-27T15:13:33Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2749 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 15:13:33.125545 10373 mini_oidc.cc:152] Starting JWKS server
I20260927 15:13:33.128065 10373 webserver.cc:460] Webserver started at https://127.0.0.1:44773/ using document root <none> and password file <none>
I20260927 15:13:33.128343 10373 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 15:13:33.128995 10373 webserver.cc:460] Webserver started at http://127.0.0.1:39177/ using document root <none> and password file <none>
2026-09-27T15:13:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:13:33Z Disabled control of system clock
I20260927 15:13:33.153954 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:32971
--webserver_interface=127.10.33.126
--webserver_port=0
--builtin_ntp_servers=127.10.33.84:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:32971
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44773/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521968392480-10373-0/testchainca.cert with env {}
W20260927 15:13:33.484149 15536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:33.484488 15536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:33.484622 15536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:33.492055 15536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:13:33.492189 15536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:33.492245 15536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:33.492323 15536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:13:33.492383 15536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:13:33.492479 15536 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:13:33.492530 15536 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44773/jwks.json
I20260927 15:13:33.503831 15536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:32971
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:32971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44773/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521968392480-10373-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:33.505250 15536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:33.507100 15536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:33.520232 15541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:33.520387 15542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:33.520447 15544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:33.521446 15536 server_base.cc:1034] running on GCE node
I20260927 15:13:33.522384 15536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:33.523938 15536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:33.525178 15536 hybrid_clock.cc:648] HybridClock initialized: now 1790522013525141 us; error 34 us; skew 500 ppm
I20260927 15:13:33.525557 15536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:13:33.527989 15536 webserver.cc:460] Webserver started at http://127.10.33.126:44637/ using document root <none> and password file <none>
I20260927 15:13:33.528710 15536 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:33.528817 15536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:33.529112 15536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:33.532172 15536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "3efc795b6cb345e181fc5df8a3b419ef"
format_stamp: "Formatted at 2026-09-27 15:13:33 on dist-test-slave-f5h6"
server_key: "e10ddec298d2e07c7809815dd92ff140"
server_key_iv: "ca5a8b712e2a35f17cf9339ba7e6fd09"
server_key_version: "encryptionkey@0"
I20260927 15:13:33.532963 15536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "3efc795b6cb345e181fc5df8a3b419ef"
format_stamp: "Formatted at 2026-09-27 15:13:33 on dist-test-slave-f5h6"
server_key: "e10ddec298d2e07c7809815dd92ff140"
server_key_iv: "ca5a8b712e2a35f17cf9339ba7e6fd09"
server_key_version: "encryptionkey@0"
I20260927 15:13:33.538200 15536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260927 15:13:33.541601 15550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:33.542999 15536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:13:33.543151 15536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "3efc795b6cb345e181fc5df8a3b419ef"
format_stamp: "Formatted at 2026-09-27 15:13:33 on dist-test-slave-f5h6"
server_key: "e10ddec298d2e07c7809815dd92ff140"
server_key_iv: "ca5a8b712e2a35f17cf9339ba7e6fd09"
server_key_version: "encryptionkey@0"
I20260927 15:13:33.543340 15536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:33.572444 15536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:33.573278 15536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:33.573536 15536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:33.590342 15536 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:32971
I20260927 15:13:33.590363 15601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:32971 every 8 connection(s)
I20260927 15:13:33.591709 15536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:13:33.593451 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 15536
I20260927 15:13:33.593631 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:13:33.594250 10373 external_mini_cluster.cc:1428] Setting key cb27f4e8b2f8ca565223ab77f305db6a
I20260927 15:13:33.596010 15602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:33.609115 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:33.596366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36676 (local address 127.10.33.126:32971)
0927 15:13:33.597345 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:33.597357 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:33.597398 (+ 41us) server_negotiation.cc:407] Connection header received
0927 15:13:33.599012 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:33.599037 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:33.599570 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:33.599928 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:33.600757 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:33.602285 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:33.604404 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:33.605138 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:33.605284 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:33.606717 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:33.606725 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:33.606742 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:13:33.607340 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:33.607425 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:13:33.607450 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:33.607552 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:33.607781 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 15:13:33.608120 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":355,"threads_started":1}
I20260927 15:13:33.610459 15602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef: Bootstrap starting.
I20260927 15:13:33.613283 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:13:33.613996 15602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:33.615016 15602 log.cc:824] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:33.617869 15602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef: No bootstrap required, opened a new log
I20260927 15:13:33.626188 15602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efc795b6cb345e181fc5df8a3b419ef" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32971 } }
I20260927 15:13:33.626740 15602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:33.626848 15602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efc795b6cb345e181fc5df8a3b419ef, State: Initialized, Role: FOLLOWER
I20260927 15:13:33.627429 15602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efc795b6cb345e181fc5df8a3b419ef" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32971 } }
I20260927 15:13:33.627658 15602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:13:33.627763 15602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:13:33.627910 15602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:33.630319 15602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efc795b6cb345e181fc5df8a3b419ef" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32971 } }
I20260927 15:13:33.630846 15602 leader_election.cc:302] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3efc795b6cb345e181fc5df8a3b419ef; no voters:
I20260927 15:13:33.631446 15602 leader_election.cc:288] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:13:33.631587 15607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:33.632309 15607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef [term 1 LEADER]: Becoming Leader. State: Replica: 3efc795b6cb345e181fc5df8a3b419ef, State: Running, Role: LEADER
I20260927 15:13:33.632849 15607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efc795b6cb345e181fc5df8a3b419ef" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32971 } }
I20260927 15:13:33.633596 15602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:13:33.635705 15608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3efc795b6cb345e181fc5df8a3b419ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efc795b6cb345e181fc5df8a3b419ef" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32971 } } }
I20260927 15:13:33.636277 15608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:33.636703 15609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3efc795b6cb345e181fc5df8a3b419ef. Latest consensus state: current_term: 1 leader_uuid: "3efc795b6cb345e181fc5df8a3b419ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efc795b6cb345e181fc5df8a3b419ef" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32971 } } }
I20260927 15:13:33.637012 15609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:33.638625 15615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:13:33.643909 15615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:13:33.653508 15615 catalog_manager.cc:1347] Generated new cluster ID: a259f3dbb18d488bad330ca80f1d296a
I20260927 15:13:33.653648 15615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:13:33.671283 15615 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:13:33.672961 15615 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:13:33.680464 15615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3efc795b6cb345e181fc5df8a3b419ef: Generated new TSK 0
I20260927 15:13:33.681378 15615 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 15:13:33.719671 15632 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 15:13:33.719890 15632 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 15:13:33.720853 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:33.698873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36690 (local address 127.10.33.126:32971)
0927 15:13:33.699109 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:33.699120 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:33.699585 (+ 465us) server_negotiation.cc:407] Connection header received
0927 15:13:33.699993 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:33.699997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:33.700124 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:33.700283 (+ 159us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:13:33.700949 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:33.701770 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:33.703896 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:33.704370 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:33.704477 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:33.705184 (+ 707us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:13:33.720314 (+ 15130us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44773/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 15:13:33.720617 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36690: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44773/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":83}
W20260927 15:13:33.720975 15605 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36690: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44773/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 15:13:33.723249 15631 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:13:33.698763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32971 (local address 127.0.0.1:36690)
0927 15:13:33.699443 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:33.699464 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:33.699688 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:33.700340 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:33.700345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:33.700359 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:13:33.700772 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:33.700784 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:33.701939 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:33.701944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:33.703730 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:33.703747 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:33.704560 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:33.704566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:33.704760 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:33.704849 (+ 89us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:13:33.720632 (+ 15783us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44773/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 15:13:33.721014 (+ 382us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.33.126:32971: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44773/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":322,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.10.33.126:32971: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44773/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 15:13:33.725169 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 15536
I20260927 15:13:33.726497 15597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:33.822463 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 15536
2026-09-27T15:13:33Z chronyd exiting
I20260927 15:13:33.869570 10373 test_util.cc:175] -----------------------------------------------
I20260927 15:13:33.869755 10373 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521968392480-10373-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (748 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 15:13:33.873505 10373 mini_oidc.cc:152] Starting JWKS server
I20260927 15:13:33.875417 10373 webserver.cc:460] Webserver started at https://127.0.0.1:38891/ using document root <none> and password file <none>
I20260927 15:13:33.875628 10373 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 15:13:33.876250 10373 webserver.cc:460] Webserver started at http://127.0.0.1:34485/ using document root <none> and password file <none>
2026-09-27T15:13:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:13:33Z Disabled control of system clock
I20260927 15:13:33.902207 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:37453
--webserver_interface=127.10.33.126
--webserver_port=0
--builtin_ntp_servers=127.10.33.84:39245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:37453
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38891/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521968392480-10373-0/testchainca.cert with env {}
W20260927 15:13:34.264441 15639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:34.264770 15639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:34.264855 15639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:34.272293 15639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:13:34.272419 15639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:34.272457 15639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:34.272511 15639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:13:34.272552 15639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:13:34.272610 15639 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:13:34.272650 15639 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38891/jwks.json
I20260927 15:13:34.283663 15639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:37453
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:37453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38891/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521968392480-10373-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:34.285040 15639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:34.286486 15639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:34.298743 15647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:34.298668 15644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:34.298750 15645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:34.300072 15639 server_base.cc:1034] running on GCE node
I20260927 15:13:34.301153 15639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:34.302968 15639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:34.304219 15639 hybrid_clock.cc:648] HybridClock initialized: now 1790522014304158 us; error 54 us; skew 500 ppm
I20260927 15:13:34.304601 15639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:13:34.307129 15639 webserver.cc:460] Webserver started at http://127.10.33.126:41501/ using document root <none> and password file <none>
I20260927 15:13:34.308012 15639 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:34.308135 15639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:34.308455 15639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:34.311513 15639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "a5b1dd5d574745f395354eea2c28eb1c"
format_stamp: "Formatted at 2026-09-27 15:13:34 on dist-test-slave-f5h6"
server_key: "38240831d9e99461b958b3d43af8fddc"
server_key_iv: "74c6bb712d39e5a1cb03997aff34d81d"
server_key_version: "encryptionkey@0"
I20260927 15:13:34.312362 15639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "a5b1dd5d574745f395354eea2c28eb1c"
format_stamp: "Formatted at 2026-09-27 15:13:34 on dist-test-slave-f5h6"
server_key: "38240831d9e99461b958b3d43af8fddc"
server_key_iv: "74c6bb712d39e5a1cb03997aff34d81d"
server_key_version: "encryptionkey@0"
I20260927 15:13:34.318472 15639 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 15:13:34.322767 15653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:34.324275 15639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:13:34.324477 15639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "a5b1dd5d574745f395354eea2c28eb1c"
format_stamp: "Formatted at 2026-09-27 15:13:34 on dist-test-slave-f5h6"
server_key: "38240831d9e99461b958b3d43af8fddc"
server_key_iv: "74c6bb712d39e5a1cb03997aff34d81d"
server_key_version: "encryptionkey@0"
I20260927 15:13:34.324734 15639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:34.355440 15639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:34.356336 15639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:34.356633 15639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:34.376334 15639 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:37453
I20260927 15:13:34.376353 15704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:37453 every 8 connection(s)
I20260927 15:13:34.378106 15639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:13:34.382155 15705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:34.383236 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 15639
I20260927 15:13:34.383430 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:13:34.384377 10373 external_mini_cluster.cc:1428] Setting key 120e221bf3c3be4b937299fe10d2d7f6
I20260927 15:13:34.396399 15705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c: Bootstrap starting.
I20260927 15:13:34.398808 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:34.386899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59192 (local address 127.10.33.126:37453)
0927 15:13:34.388492 (+ 1593us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:34.388509 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:34.388581 (+ 72us) server_negotiation.cc:407] Connection header received
0927 15:13:34.390320 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:34.390346 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:34.390883 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:34.391270 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:34.391828 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:34.393079 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:34.394412 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:34.395125 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:34.395294 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:34.396447 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:34.396456 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:34.396472 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:13:34.397062 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:34.397150 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:13:34.397172 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:34.397276 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:34.397491 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 15:13:34.397814 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":811,"thread_start_us":466,"threads_started":1}
I20260927 15:13:34.399688 15705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:34.400705 15705 log.cc:824] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:34.402627 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:13:34.403769 15705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c: No bootstrap required, opened a new log
I20260927 15:13:34.411659 15705 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b1dd5d574745f395354eea2c28eb1c" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 37453 } }
I20260927 15:13:34.412034 15705 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:34.412130 15705 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b1dd5d574745f395354eea2c28eb1c, State: Initialized, Role: FOLLOWER
I20260927 15:13:34.412685 15705 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b1dd5d574745f395354eea2c28eb1c" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 37453 } }
I20260927 15:13:34.412914 15705 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:13:34.413031 15705 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:13:34.413190 15705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:34.415414 15705 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b1dd5d574745f395354eea2c28eb1c" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 37453 } }
I20260927 15:13:34.415923 15705 leader_election.cc:302] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5b1dd5d574745f395354eea2c28eb1c; no voters:
I20260927 15:13:34.416497 15705 leader_election.cc:288] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:13:34.416631 15710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:34.417326 15710 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c [term 1 LEADER]: Becoming Leader. State: Replica: a5b1dd5d574745f395354eea2c28eb1c, State: Running, Role: LEADER
I20260927 15:13:34.417909 15710 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b1dd5d574745f395354eea2c28eb1c" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 37453 } }
I20260927 15:13:34.418583 15705 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:13:34.421128 15711 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5b1dd5d574745f395354eea2c28eb1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b1dd5d574745f395354eea2c28eb1c" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 37453 } } }
I20260927 15:13:34.421171 15712 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5b1dd5d574745f395354eea2c28eb1c. Latest consensus state: current_term: 1 leader_uuid: "a5b1dd5d574745f395354eea2c28eb1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b1dd5d574745f395354eea2c28eb1c" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 37453 } } }
I20260927 15:13:34.421610 15712 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:34.423103 15716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:13:34.423122 15711 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:34.429227 15716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:13:34.440124 15716 catalog_manager.cc:1347] Generated new cluster ID: b73ae1075a284da29c0dc644afbd1bad
I20260927 15:13:34.440239 15716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:13:34.450270 15716 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:13:34.451567 15716 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:13:34.457995 15716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5b1dd5d574745f395354eea2c28eb1c: Generated new TSK 0
I20260927 15:13:34.459432 15716 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 15:13:34.502061 15735 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 15:13:34.502188 15735 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 15:13:34.503520 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:34.481642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59206 (local address 127.10.33.126:37453)
0927 15:13:34.481841 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:34.481847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:34.482199 (+ 352us) server_negotiation.cc:407] Connection header received
0927 15:13:34.482413 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:34.482419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:34.482547 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:34.482754 (+ 207us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:13:34.483623 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:34.484768 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:34.486329 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:34.487011 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:34.487152 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:34.487720 (+ 568us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:13:34.502809 (+ 15089us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 15:13:34.503197 (+ 388us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59206: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 15:13:34.503631 15734 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:13:34.481401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:37453 (local address 127.0.0.1:59206)
0927 15:13:34.482047 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:34.482107 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:34.482234 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:34.482900 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:34.482906 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:34.482924 (+ 18us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:13:34.483412 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:34.483431 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:34.484947 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:34.484952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:34.486181 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:34.486195 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:34.487222 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:34.487227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:34.487376 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:34.487417 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:13:34.503097 (+ 15680us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 15:13:34.503327 (+ 230us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.33.126:37453: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":363,"threads_started":1}
W20260927 15:13:34.503659 15708 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59206: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260927 15:13:34.504884 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 15639
I20260927 15:13:34.506093 15700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:34.598889 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 15639
2026-09-27T15:13:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (778 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 15:13:34.652053 10373 mini_oidc.cc:152] Starting JWKS server
I20260927 15:13:34.654395 10373 webserver.cc:460] Webserver started at https://127.0.0.1:33947/ using document root <none> and password file <none>
I20260927 15:13:34.654647 10373 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 15:13:34.655263 10373 webserver.cc:460] Webserver started at http://127.0.0.1:33853/ using document root <none> and password file <none>
2026-09-27T15:13:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:13:34Z Disabled control of system clock
I20260927 15:13:34.680663 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:42227
--webserver_interface=127.10.33.126
--webserver_port=0
--builtin_ntp_servers=127.10.33.84:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:42227
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33947/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 15:13:35.015318 15742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:35.015674 15742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:35.015793 15742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:35.023216 15742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:13:35.023360 15742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:35.023414 15742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:35.023492 15742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:13:35.023550 15742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:13:35.023634 15742 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:13:35.023686 15742 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33947/jwks.json
I20260927 15:13:35.033304 15742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:42227
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:42227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33947/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:35.034672 15742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:35.036458 15742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:35.048266 15747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:35.048373 15750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:35.048411 15748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:35.049461 15742 server_base.cc:1034] running on GCE node
I20260927 15:13:35.050400 15742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:35.052587 15742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:35.053830 15742 hybrid_clock.cc:648] HybridClock initialized: now 1790522015053772 us; error 55 us; skew 500 ppm
I20260927 15:13:35.054296 15742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:13:35.056831 15742 webserver.cc:460] Webserver started at http://127.10.33.126:40133/ using document root <none> and password file <none>
I20260927 15:13:35.057562 15742 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:35.057667 15742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:35.058039 15742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:35.061175 15742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "6651f2b3d1d0424b87bc09722acf5c69"
format_stamp: "Formatted at 2026-09-27 15:13:35 on dist-test-slave-f5h6"
server_key: "f52c4c7b898b5e701c330cb29e446847"
server_key_iv: "42f999c82934d1bc30d17e006295c26d"
server_key_version: "encryptionkey@0"
I20260927 15:13:35.062136 15742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "6651f2b3d1d0424b87bc09722acf5c69"
format_stamp: "Formatted at 2026-09-27 15:13:35 on dist-test-slave-f5h6"
server_key: "f52c4c7b898b5e701c330cb29e446847"
server_key_iv: "42f999c82934d1bc30d17e006295c26d"
server_key_version: "encryptionkey@0"
I20260927 15:13:35.067441 15742 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:13:35.071036 15756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:35.072469 15742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:13:35.072670 15742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "6651f2b3d1d0424b87bc09722acf5c69"
format_stamp: "Formatted at 2026-09-27 15:13:35 on dist-test-slave-f5h6"
server_key: "f52c4c7b898b5e701c330cb29e446847"
server_key_iv: "42f999c82934d1bc30d17e006295c26d"
server_key_version: "encryptionkey@0"
I20260927 15:13:35.072883 15742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:35.093385 15742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:35.094830 15742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:35.095187 15742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:35.114986 15742 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:42227
I20260927 15:13:35.114962 15807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:42227 every 8 connection(s)
I20260927 15:13:35.116334 15742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:13:35.120180 15808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:35.120173 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 15742
I20260927 15:13:35.120347 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:13:35.120929 10373 external_mini_cluster.cc:1428] Setting key df066651a3a1745a36192698b46e426d
I20260927 15:13:35.134183 15808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69: Bootstrap starting.
I20260927 15:13:35.138075 15808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:35.138485 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:35.122725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41820 (local address 127.10.33.126:42227)
0927 15:13:35.124373 (+ 1648us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:35.124392 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:35.124462 (+ 70us) server_negotiation.cc:407] Connection header received
0927 15:13:35.126833 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:35.126874 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:35.127617 (+ 743us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:35.128130 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:35.129113 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:35.130611 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:35.132196 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:35.133300 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:35.133603 (+ 303us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:35.135715 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:35.135731 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:35.135750 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:13:35.136432 (+ 682us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:35.136544 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:13:35.136573 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:35.136709 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:35.136988 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 15:13:35.137357 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":536,"threads_started":1}
I20260927 15:13:35.139392 15808 log.cc:824] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:35.143612 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:13:35.144392 15808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69: No bootstrap required, opened a new log
I20260927 15:13:35.152441 15808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6651f2b3d1d0424b87bc09722acf5c69" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42227 } }
I20260927 15:13:35.152872 15808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:35.152962 15808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6651f2b3d1d0424b87bc09722acf5c69, State: Initialized, Role: FOLLOWER
I20260927 15:13:35.153565 15808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6651f2b3d1d0424b87bc09722acf5c69" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42227 } }
I20260927 15:13:35.153787 15808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:13:35.153965 15808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:13:35.154127 15808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:35.156265 15808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6651f2b3d1d0424b87bc09722acf5c69" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42227 } }
I20260927 15:13:35.156792 15808 leader_election.cc:302] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6651f2b3d1d0424b87bc09722acf5c69; no voters:
I20260927 15:13:35.157440 15808 leader_election.cc:288] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:13:35.157593 15813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:35.158506 15813 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69 [term 1 LEADER]: Becoming Leader. State: Replica: 6651f2b3d1d0424b87bc09722acf5c69, State: Running, Role: LEADER
I20260927 15:13:35.159132 15813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6651f2b3d1d0424b87bc09722acf5c69" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42227 } }
I20260927 15:13:35.159693 15808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:13:35.162083 15814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6651f2b3d1d0424b87bc09722acf5c69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6651f2b3d1d0424b87bc09722acf5c69" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42227 } } }
I20260927 15:13:35.162050 15815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6651f2b3d1d0424b87bc09722acf5c69. Latest consensus state: current_term: 1 leader_uuid: "6651f2b3d1d0424b87bc09722acf5c69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6651f2b3d1d0424b87bc09722acf5c69" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42227 } } }
I20260927 15:13:35.162484 15814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:35.162529 15815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:35.172664 15818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:13:35.178715 15818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:13:35.187523 15818 catalog_manager.cc:1347] Generated new cluster ID: 6fab52b0d90542e3a65a01e09eaf54b0
I20260927 15:13:35.187651 15818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:13:35.201032 15818 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:13:35.202996 15818 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:13:35.219789 15818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6651f2b3d1d0424b87bc09722acf5c69: Generated new TSK 0
I20260927 15:13:35.220949 15818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:13:35.263270 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:35.255509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41828 (local address 127.10.33.126:42227)
0927 15:13:35.255711 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:35.255723 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:35.256153 (+ 430us) server_negotiation.cc:407] Connection header received
0927 15:13:35.256435 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:35.256439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:35.256523 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:35.256669 (+ 146us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:13:35.257188 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:35.258002 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:35.259367 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:35.259957 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:35.260111 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:35.260664 (+ 553us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:13:35.262630 (+ 1966us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33947/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 15:13:35.262943 (+ 313us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41828: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33947/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 15:13:35.263468 15837 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:13:35.255424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:42227 (local address 127.0.0.1:41828)
0927 15:13:35.256044 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:35.256062 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:35.256240 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:35.256699 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:35.256705 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:35.256717 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:13:35.256991 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:35.256999 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:35.258149 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:35.258154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:35.259223 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:35.259233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:35.260165 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:35.260171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:35.260326 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:35.260365 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:13:35.262836 (+ 2471us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33947/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 15:13:35.263084 (+ 248us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.33.126:42227: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33947/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":330,"threads_started":1}
W20260927 15:13:35.263715 15811 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41828: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33947/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260927 15:13:35.264611 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 15742
I20260927 15:13:35.266036 15803 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 15:13:35.275815 15838 webserver.cc:533] Webserver: sslize failed:
E20260927 15:13:35.275941 15838 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 15:13:35.361724 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 15742
2026-09-27T15:13:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (763 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T15:13:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:13:35Z Disabled control of system clock
I20260927 15:13:35.443516 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:44459
--webserver_interface=127.10.33.126
--webserver_port=0
--builtin_ntp_servers=127.10.33.84:45341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:44459
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521968392480-10373-0/testchainca.cert with env {}
W20260927 15:13:35.799278 15843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:35.799602 15843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:35.799717 15843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:35.808007 15843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:13:35.808161 15843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:35.808198 15843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:35.808274 15843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:13:35.808331 15843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:13:35.808419 15843 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:13:35.808468 15843 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 15:13:35.819267 15843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:45341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:44459
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:44459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521968392480-10373-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:35.820622 15843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:35.821959 15843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:35.834362 15849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:35.834380 15848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:35.834523 15851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:35.835209 15843 server_base.cc:1034] running on GCE node
I20260927 15:13:35.836040 15843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:35.837579 15843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:35.838783 15843 hybrid_clock.cc:648] HybridClock initialized: now 1790522015838715 us; error 66 us; skew 500 ppm
I20260927 15:13:35.839175 15843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:13:35.841478 15843 webserver.cc:460] Webserver started at http://127.10.33.126:39271/ using document root <none> and password file <none>
I20260927 15:13:35.842209 15843 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:35.842314 15843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:35.842602 15843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:35.845546 15843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "b9ee56d45dee4af493d00bd5548238aa"
format_stamp: "Formatted at 2026-09-27 15:13:35 on dist-test-slave-f5h6"
server_key: "f9365b0e9545108816135199195c8fc6"
server_key_iv: "46db046e1266d4b53728545cdf3d1704"
server_key_version: "encryptionkey@0"
I20260927 15:13:35.846369 15843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "b9ee56d45dee4af493d00bd5548238aa"
format_stamp: "Formatted at 2026-09-27 15:13:35 on dist-test-slave-f5h6"
server_key: "f9365b0e9545108816135199195c8fc6"
server_key_iv: "46db046e1266d4b53728545cdf3d1704"
server_key_version: "encryptionkey@0"
I20260927 15:13:35.851519 15843 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:13:35.855377 15857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:35.856832 15843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:13:35.857020 15843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "b9ee56d45dee4af493d00bd5548238aa"
format_stamp: "Formatted at 2026-09-27 15:13:35 on dist-test-slave-f5h6"
server_key: "f9365b0e9545108816135199195c8fc6"
server_key_iv: "46db046e1266d4b53728545cdf3d1704"
server_key_version: "encryptionkey@0"
I20260927 15:13:35.857273 15843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:35.869805 15843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:35.870926 15843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:35.871240 15843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:35.890446 15843 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:44459
I20260927 15:13:35.890551 15908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:44459 every 8 connection(s)
I20260927 15:13:35.891954 15843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:13:35.895068 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 15843
I20260927 15:13:35.895267 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:13:35.895859 10373 external_mini_cluster.cc:1428] Setting key d31c7124bf6f3aa23c397bb33376a5ec
I20260927 15:13:35.895953 15909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:35.909096 15909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa: Bootstrap starting.
I20260927 15:13:35.910115 15912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:35.897720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36258 (local address 127.10.33.126:44459)
0927 15:13:35.898883 (+ 1163us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:35.898898 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:35.898952 (+ 54us) server_negotiation.cc:407] Connection header received
0927 15:13:35.900826 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:35.900857 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:35.901468 (+ 611us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:35.901947 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:35.902586 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:35.903894 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:35.905240 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:35.906040 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:35.906290 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:35.907627 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:35.907635 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:35.907653 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:13:35.908226 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:35.908316 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:13:35.908341 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:35.908455 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:35.908694 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 15:13:35.909025 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":391,"threads_started":1}
I20260927 15:13:35.912489 15909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:35.913666 15909 log.cc:824] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:35.914057 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:13:35.916584 15909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa: No bootstrap required, opened a new log
I20260927 15:13:35.924649 15909 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ee56d45dee4af493d00bd5548238aa" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44459 } }
I20260927 15:13:35.925048 15909 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:35.925125 15909 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ee56d45dee4af493d00bd5548238aa, State: Initialized, Role: FOLLOWER
I20260927 15:13:35.925674 15909 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ee56d45dee4af493d00bd5548238aa" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44459 } }
I20260927 15:13:35.925920 15909 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:13:35.926029 15909 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:13:35.926162 15909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:35.928222 15909 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ee56d45dee4af493d00bd5548238aa" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44459 } }
I20260927 15:13:35.928682 15909 leader_election.cc:302] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9ee56d45dee4af493d00bd5548238aa; no voters:
I20260927 15:13:35.929296 15909 leader_election.cc:288] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:13:35.929450 15914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:35.930152 15914 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa [term 1 LEADER]: Becoming Leader. State: Replica: b9ee56d45dee4af493d00bd5548238aa, State: Running, Role: LEADER
I20260927 15:13:35.930773 15914 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ee56d45dee4af493d00bd5548238aa" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44459 } }
I20260927 15:13:35.931588 15909 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:13:35.933456 15916 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9ee56d45dee4af493d00bd5548238aa. Latest consensus state: current_term: 1 leader_uuid: "b9ee56d45dee4af493d00bd5548238aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ee56d45dee4af493d00bd5548238aa" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44459 } } }
I20260927 15:13:35.933900 15916 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:35.936265 15920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:13:35.933466 15915 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9ee56d45dee4af493d00bd5548238aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ee56d45dee4af493d00bd5548238aa" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44459 } } }
I20260927 15:13:35.936863 15915 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:35.941823 15920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:13:35.951431 15920 catalog_manager.cc:1347] Generated new cluster ID: c8880a373ea8464bbc797126636b3197
I20260927 15:13:35.951571 15920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:13:35.968871 15920 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:13:35.970336 15920 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:13:35.977375 15920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9ee56d45dee4af493d00bd5548238aa: Generated new TSK 0
I20260927 15:13:35.978359 15920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:13:36.085721 15912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:35.996249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36268 (local address 127.10.33.126:44459)
0927 15:13:35.996449 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:35.996455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:35.997816 (+ 1361us) server_negotiation.cc:407] Connection header received
0927 15:13:35.998095 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:35.998101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:35.998205 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:35.998332 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:13:35.998870 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:35.999656 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:36.001680 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:36.002222 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:36.002319 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:36.002878 (+ 559us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:13:36.085141 (+ 82263us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 15:13:36.085485 (+ 344us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36268: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 15:13:36.085878 15912 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36268: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 15:13:36.086261 15938 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:13:35.996567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:44459 (local address 127.0.0.1:36268)
0927 15:13:35.997690 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:35.997709 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:35.997914 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:35.998375 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:35.998381 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:35.998394 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:13:35.998711 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:35.998723 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:35.999801 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:35.999806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:36.001517 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:36.001533 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:36.002391 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:36.002398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:36.002582 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:36.002632 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:13:36.085433 (+ 82801us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 15:13:36.085749 (+ 316us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.33.126:44459: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":973,"thread_start_us":819,"threads_started":1}
I20260927 15:13:36.087703 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 15843
I20260927 15:13:36.089030 15904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:36.192199 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 15843
2026-09-27T15:13:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (808 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (409 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (425 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:13:37 dist-test-slave-f5h6 krb5kdc[15953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:13:37 dist-test-slave-f5h6 krb5kdc[15953](info): set up 2 sockets
Sep 27 15:13:37 dist-test-slave-f5h6 krb5kdc[15953](info): commencing operation
krb5kdc: starting...
W20260927 15:13:38.050415 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.006s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:13:38 dist-test-slave-f5h6 krb5kdc[15953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:13:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:13:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:13:38.272195 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:32993
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:32993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:38.590728 15966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:38.591048 15966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:38.591138 15966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:38.598565 15966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:13:38.598699 15966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:38.598743 15966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:38.598804 15966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:13:38.598868 15966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:13:38.609766 15966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:32993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:32993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:38.611205 15966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:38.612596 15966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:38.624697 15972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:38.624625 15971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:38.624719 15974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:38.626101 15966 server_base.cc:1034] running on GCE node
I20260927 15:13:38.626958 15966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:38.628444 15966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:38.629679 15966 hybrid_clock.cc:648] HybridClock initialized: now 1790522018629622 us; error 46 us; skew 500 ppm
I20260927 15:13:38.630076 15966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:38 dist-test-slave-f5h6 krb5kdc[15953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522018, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:38.633236 15966 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:13:38.634771 15966 webserver.cc:460] Webserver started at http://127.10.33.126:43251/ using document root <none> and password file <none>
I20260927 15:13:38.635438 15966 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:38.635545 15966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:38.635828 15966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:38.638827 15966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "f4ce660ec11e46e89e88cd6176460b99"
format_stamp: "Formatted at 2026-09-27 15:13:38 on dist-test-slave-f5h6"
server_key: "ed6ccd394102adb314b7b8d1f94b1140"
server_key_iv: "da340d1f7039fee5a00a5b2a570b4548"
server_key_version: "encryptionkey@0"
I20260927 15:13:38.639565 15966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "f4ce660ec11e46e89e88cd6176460b99"
format_stamp: "Formatted at 2026-09-27 15:13:38 on dist-test-slave-f5h6"
server_key: "ed6ccd394102adb314b7b8d1f94b1140"
server_key_iv: "da340d1f7039fee5a00a5b2a570b4548"
server_key_version: "encryptionkey@0"
I20260927 15:13:38.644785 15966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 15:13:38.648507 15981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:38.649997 15966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:13:38.650180 15966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "f4ce660ec11e46e89e88cd6176460b99"
format_stamp: "Formatted at 2026-09-27 15:13:38 on dist-test-slave-f5h6"
server_key: "ed6ccd394102adb314b7b8d1f94b1140"
server_key_iv: "da340d1f7039fee5a00a5b2a570b4548"
server_key_version: "encryptionkey@0"
I20260927 15:13:38.650398 15966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:38.664448 15966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:38.668819 15966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:38.669072 15966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:38.687040 15966 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:32993
I20260927 15:13:38.687085 16032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:32993 every 8 connection(s)
I20260927 15:13:38.688441 15966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:13:38.691622 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 15966
I20260927 15:13:38.691833 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:13:38.692556 16033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:38.692589 10373 external_mini_cluster.cc:1428] Setting key c746e7136b2887993e9d92fbd3613b6a
I20260927 15:13:38.707278 16033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99: Bootstrap starting.
Sep 27 15:13:38 dist-test-slave-f5h6 krb5kdc[15953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:38.711126 16033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:38.712281 16033 log.cc:824] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:38.716490 16033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99: No bootstrap required, opened a new log
I20260927 15:13:38.716939 16036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:38.694955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42240 (local address 127.10.33.126:32993)
0927 15:13:38.695761 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:38.695773 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:38.695817 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:13:38.697329 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:38.697350 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:38.697740 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:38.698098 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:38.699859 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:38.701229 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:38.702958 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:38.703737 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:38.703881 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:38.708544 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:38.708554 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:38.708570 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:38.708619 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:38.711757 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:38.712624 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:38.712635 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:38.712644 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:38.712817 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:38.713214 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:38.713218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:38.713220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:38.714527 (+ 1307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:38.714860 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:38.715372 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 15:13:38.715842 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":377,"threads_started":1}
I20260927 15:13:38.725230 16033 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ce660ec11e46e89e88cd6176460b99" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32993 } }
I20260927 15:13:38.725637 16033 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:38.725726 16033 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4ce660ec11e46e89e88cd6176460b99, State: Initialized, Role: FOLLOWER
I20260927 15:13:38.726353 16033 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ce660ec11e46e89e88cd6176460b99" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32993 } }
I20260927 15:13:38.726614 16033 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:13:38.726760 16033 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:13:38.726922 16033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:38.729017 16033 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ce660ec11e46e89e88cd6176460b99" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32993 } }
I20260927 15:13:38.729503 16033 leader_election.cc:302] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4ce660ec11e46e89e88cd6176460b99; no voters:
I20260927 15:13:38.730158 16033 leader_election.cc:288] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:13:38.730278 16038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:38.730894 16038 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99 [term 1 LEADER]: Becoming Leader. State: Replica: f4ce660ec11e46e89e88cd6176460b99, State: Running, Role: LEADER
I20260927 15:13:38.731417 16038 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ce660ec11e46e89e88cd6176460b99" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32993 } }
I20260927 15:13:38.732131 16033 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:13:38.734220 16039 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4ce660ec11e46e89e88cd6176460b99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ce660ec11e46e89e88cd6176460b99" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32993 } } }
I20260927 15:13:38.734642 16039 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:38.734048 16040 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4ce660ec11e46e89e88cd6176460b99. Latest consensus state: current_term: 1 leader_uuid: "f4ce660ec11e46e89e88cd6176460b99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ce660ec11e46e89e88cd6176460b99" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32993 } } }
I20260927 15:13:38.736326 16040 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99 [sys.catalog]: This master's current role is: LEADER
W20260927 15:13:38.743489 16054 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:13:38.743664 16054 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:13:38.743804 16044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:13:38.747160 16044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:13:38.757004 16044 catalog_manager.cc:1347] Generated new cluster ID: fd88da9814fc484bbec609cd9ed9543f
I20260927 15:13:38.757138 16044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:13:38.798766 16044 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:13:38.800673 16044 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:13:38.813670 16044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4ce660ec11e46e89e88cd6176460b99: Generated new TSK 0
I20260927 15:13:38.814832 16044 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.33.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
I20260927 15:13:38.866641 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:32993
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:39.197010 16059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:39.197311 16059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:39.197438 16059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:39.204802 16059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:39.204942 16059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:39.205102 16059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:13:39.217315 16059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:32993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:39.218742 16059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:39.220021 16059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:39.233934 16065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:39.234011 16067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:39.234102 16064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:39.234894 16059 server_base.cc:1034] running on GCE node
I20260927 15:13:39.235554 16059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:39.236667 16059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:39.237927 16059 hybrid_clock.cc:648] HybridClock initialized: now 1790522019237845 us; error 60 us; skew 500 ppm
I20260927 15:13:39.238276 16059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:39 dist-test-slave-f5h6 krb5kdc[15953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522019, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:39.241827 16059 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:13:39.243438 16059 webserver.cc:460] Webserver started at http://127.10.33.65:34213/ using document root <none> and password file <none>
I20260927 15:13:39.244156 16059 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:39.244280 16059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:39.244570 16059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:39.247686 16059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "bbb0166938ee495285f0351b8f0a911e"
format_stamp: "Formatted at 2026-09-27 15:13:39 on dist-test-slave-f5h6"
server_key: "2af41a4ac4b76d769d00a92109cfcb15"
server_key_iv: "fc2de94406801e68550bd18ab09d3fb2"
server_key_version: "encryptionkey@0"
I20260927 15:13:39.248495 16059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "bbb0166938ee495285f0351b8f0a911e"
format_stamp: "Formatted at 2026-09-27 15:13:39 on dist-test-slave-f5h6"
server_key: "2af41a4ac4b76d769d00a92109cfcb15"
server_key_iv: "fc2de94406801e68550bd18ab09d3fb2"
server_key_version: "encryptionkey@0"
I20260927 15:13:39.253669 16059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 15:13:39.257452 16074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:39.259078 16059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:13:39.259256 16059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "bbb0166938ee495285f0351b8f0a911e"
format_stamp: "Formatted at 2026-09-27 15:13:39 on dist-test-slave-f5h6"
server_key: "2af41a4ac4b76d769d00a92109cfcb15"
server_key_iv: "fc2de94406801e68550bd18ab09d3fb2"
server_key_version: "encryptionkey@0"
I20260927 15:13:39.259464 16059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:39.285104 16059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:39.288985 16059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:39.289247 16059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:39.290165 16059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:39.291395 16059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:39.291471 16059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:39.291579 16059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:39.291631 16059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:39.320556 16059 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:35925
I20260927 15:13:39.320601 16186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:35925 every 8 connection(s)
I20260927 15:13:39.321830 16059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:13:39.328513 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 16059
I20260927 15:13:39.328675 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:13:39.329276 10373 external_mini_cluster.cc:1428] Setting key 00de3060ee9d475cb72a830b23e5e13f
Sep 27 15:13:39 dist-test-slave-f5h6 krb5kdc[15953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522019, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:39.342079 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:39.324350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:36789 (local address 127.10.33.126:32993)
0927 15:13:39.324781 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:39.324785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:39.325576 (+ 791us) server_negotiation.cc:407] Connection header received
0927 15:13:39.327343 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:39.327348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:39.327432 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:39.327522 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:39.328967 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:39.329705 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:39.331905 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:39.332369 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:39.332472 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:39.336297 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:39.336302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:39.336305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:39.336337 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:39.338773 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:39.339648 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:39.339657 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:39.339660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:39.339828 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:39.340311 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:39.340316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:39.340319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:39.340631 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:39.340755 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:39.341592 (+ 837us) server_negotiation.cc:299] Negotiation successful
0927 15:13:39.341767 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":207,"threads_started":1}
I20260927 15:13:39.343433 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:39.324760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32993 (local address 127.10.33.65:36789)
0927 15:13:39.325422 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:39.325463 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:39.327104 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:39.327744 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:39.327752 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:39.328235 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:39.328814 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:39.328834 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:39.329912 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:39.329916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:39.331724 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:39.331740 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:39.332598 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:39.332605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:39.332902 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:39.333629 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:39.333655 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:39.336035 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:39.338901 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:39.338911 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:39.338928 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:39.339459 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:39.339932 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:39.339937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:39.339940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:39.340170 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:39.340762 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:39.340770 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:39.341185 (+ 415us) client_negotiation.cc:769] Sending connection context
0927 15:13:39.341460 (+ 275us) client_negotiation.cc:240] Negotiation successful
0927 15:13:39.342013 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":266,"threads_started":1}
I20260927 15:13:39.345059 16187 heartbeater.cc:347] Connected to a master server at 127.10.33.126:32993
I20260927 15:13:39.345395 16187 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:39.346251 16187 heartbeater.cc:511] Master 127.10.33.126:32993 requested a full tablet report, sending...
I20260927 15:13:39.348546 15998 ts_manager.cc:194] Registered new tserver with Master: bbb0166938ee495285f0351b8f0a911e (127.10.33.65:35925)
WARNING: no policy specified for kudu/127.10.33.66@KRBTEST.COM; defaulting to no policy
I20260927 15:13:39.350010 15998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:36789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
I20260927 15:13:39.380893 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:32993
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:39.720407 16193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:39.720729 16193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:39.720849 16193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:39.728684 16193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:39.728814 16193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:39.728964 16193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:13:39.739203 16193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:32993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:39.740505 16193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:39.741806 16193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:39.755595 16201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:39.755708 16198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:39.755697 16199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:39.756351 16193 server_base.cc:1034] running on GCE node
I20260927 15:13:39.757507 16193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:39.758868 16193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:39.760119 16193 hybrid_clock.cc:648] HybridClock initialized: now 1790522019760048 us; error 73 us; skew 500 ppm
I20260927 15:13:39.760483 16193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:39 dist-test-slave-f5h6 krb5kdc[15953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522019, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:39.764001 16193 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:13:39.765960 16193 webserver.cc:460] Webserver started at http://127.10.33.66:39919/ using document root <none> and password file <none>
I20260927 15:13:39.766671 16193 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:39.766790 16193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:39.767112 16193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:39.770244 16193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "3fe800c17ed445ab8116bf94f563995f"
format_stamp: "Formatted at 2026-09-27 15:13:39 on dist-test-slave-f5h6"
server_key: "59c264b4cf27f842ba9f3b39eff7950b"
server_key_iv: "7ca5dfc1c1d6edb12e22175f8538d33f"
server_key_version: "encryptionkey@0"
I20260927 15:13:39.771035 16193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "3fe800c17ed445ab8116bf94f563995f"
format_stamp: "Formatted at 2026-09-27 15:13:39 on dist-test-slave-f5h6"
server_key: "59c264b4cf27f842ba9f3b39eff7950b"
server_key_iv: "7ca5dfc1c1d6edb12e22175f8538d33f"
server_key_version: "encryptionkey@0"
I20260927 15:13:39.776368 16193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 15:13:39.780190 16208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:39.781778 16193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:13:39.782018 16193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "3fe800c17ed445ab8116bf94f563995f"
format_stamp: "Formatted at 2026-09-27 15:13:39 on dist-test-slave-f5h6"
server_key: "59c264b4cf27f842ba9f3b39eff7950b"
server_key_iv: "7ca5dfc1c1d6edb12e22175f8538d33f"
server_key_version: "encryptionkey@0"
I20260927 15:13:39.782229 16193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:39.800122 16193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:39.806353 16193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:39.806653 16193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:39.807672 16193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:39.809041 16193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:39.809118 16193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:39.809244 16193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:39.809300 16193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:39.843763 16193 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:39013
I20260927 15:13:39.843792 16320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:39013 every 8 connection(s)
I20260927 15:13:39.845126 16193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:13:39.851161 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 16193
I20260927 15:13:39.851325 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:13:39.851743 10373 external_mini_cluster.cc:1428] Setting key 73e84e9ee50dd26890b51113c5ddbf21
Sep 27 15:13:39 dist-test-slave-f5h6 krb5kdc[15953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522019, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:39.869037 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:39.847708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:60037 (local address 127.10.33.126:32993)
0927 15:13:39.848217 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:39.848221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:39.849145 (+ 924us) server_negotiation.cc:407] Connection header received
0927 15:13:39.850995 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:39.851001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:39.851101 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:39.851192 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:39.852908 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:39.853720 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:39.855383 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:39.855859 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:39.855972 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:39.861022 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:39.861041 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:39.861046 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:39.861101 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:39.864959 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:39.866155 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:39.866160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:39.866163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:39.866320 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:39.866904 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:39.866918 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:39.866922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:39.867379 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:39.867522 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:39.868579 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0927 15:13:39.868777 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":288,"threads_started":1}
I20260927 15:13:39.870671 16324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:39.848158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32993 (local address 127.10.33.66:60037)
0927 15:13:39.848921 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:39.848959 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:39.850679 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:39.851479 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:39.851488 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:39.852042 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:39.852741 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:39.852761 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:39.853896 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:39.853902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:39.855217 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:39.855229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:39.856087 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:39.856095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:39.856325 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:39.857031 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:39.857057 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:39.860383 (+ 3326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:39.865203 (+ 4820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:39.865214 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:39.865238 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:39.865948 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:39.866441 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:39.866447 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:39.866450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:39.866747 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:39.867531 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:39.867540 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:39.868075 (+ 535us) client_negotiation.cc:769] Sending connection context
0927 15:13:39.868420 (+ 345us) client_negotiation.cc:240] Negotiation successful
0927 15:13:39.869049 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":327,"threads_started":1}
WARNING: no policy specified for kudu/127.10.33.67@KRBTEST.COM; defaulting to no policy
I20260927 15:13:39.872304 16321 heartbeater.cc:347] Connected to a master server at 127.10.33.126:32993
I20260927 15:13:39.872630 16321 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:39.873395 16321 heartbeater.cc:511] Master 127.10.33.126:32993 requested a full tablet report, sending...
I20260927 15:13:39.875020 15998 ts_manager.cc:194] Registered new tserver with Master: 3fe800c17ed445ab8116bf94f563995f (127.10.33.66:39013)
I20260927 15:13:39.875874 15998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:60037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
I20260927 15:13:39.910703 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:32993
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:40.266533 16327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:40.266832 16327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:40.266953 16327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:40.274291 16327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:40.274432 16327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:40.274582 16327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:13:40.284807 16327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:32993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:40.286195 16327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:40.287549 16327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:40.301455 16333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:40.301479 16335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:40.301505 16332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:40.308600 16327 server_base.cc:1034] running on GCE node
I20260927 15:13:40.309394 16327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:40.310556 16327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:40.311844 16327 hybrid_clock.cc:648] HybridClock initialized: now 1790522020311756 us; error 59 us; skew 500 ppm
I20260927 15:13:40.312237 16327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:40 dist-test-slave-f5h6 krb5kdc[15953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522020, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:40.316484 16327 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:13:40.318271 16327 webserver.cc:460] Webserver started at http://127.10.33.67:41921/ using document root <none> and password file <none>
I20260927 15:13:40.319038 16327 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:40.319175 16327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:40.319476 16327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:40.322582 16327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "91b7988fb71f4565b4b214412da2a113"
format_stamp: "Formatted at 2026-09-27 15:13:40 on dist-test-slave-f5h6"
server_key: "08c2b730e929f19a9c8ac8b0bf4f9770"
server_key_iv: "338b5cc09be29e2b3819e3177f869e20"
server_key_version: "encryptionkey@0"
I20260927 15:13:40.323361 16327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "91b7988fb71f4565b4b214412da2a113"
format_stamp: "Formatted at 2026-09-27 15:13:40 on dist-test-slave-f5h6"
server_key: "08c2b730e929f19a9c8ac8b0bf4f9770"
server_key_iv: "338b5cc09be29e2b3819e3177f869e20"
server_key_version: "encryptionkey@0"
I20260927 15:13:40.329039 16327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:13:40.333179 16342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:40.334945 16327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:13:40.335147 16327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "91b7988fb71f4565b4b214412da2a113"
format_stamp: "Formatted at 2026-09-27 15:13:40 on dist-test-slave-f5h6"
server_key: "08c2b730e929f19a9c8ac8b0bf4f9770"
server_key_iv: "338b5cc09be29e2b3819e3177f869e20"
server_key_version: "encryptionkey@0"
I20260927 15:13:40.335371 16327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:40.353902 16187 heartbeater.cc:503] Master 127.10.33.126:32993 was elected leader, sending a full tablet report...
I20260927 15:13:40.363592 16327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:40.368003 16327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:40.368259 16327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:40.369194 16327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:40.370599 16327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:40.370671 16327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:40.370754 16327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:40.370795 16327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:40.404636 16327 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:43171
I20260927 15:13:40.404695 16454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:43171 every 8 connection(s)
I20260927 15:13:40.406389 16327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:13:40.414315 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 16327
I20260927 15:13:40.414482 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:13:40.414911 10373 external_mini_cluster.cc:1428] Setting key 22e89d1ac303dbb0b6a0e29a9565bd5a
Sep 27 15:13:40 dist-test-slave-f5h6 krb5kdc[15953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522020, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:13:40.429477 16457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:40.409678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:44725 (local address 127.10.33.126:32993)
0927 15:13:40.410175 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:40.410180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:40.411199 (+ 1019us) server_negotiation.cc:407] Connection header received
0927 15:13:40.413771 (+ 2572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:40.413778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:40.413931 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:40.414057 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:40.415935 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:40.416948 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:40.419161 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:40.419560 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:40.419673 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:40.423619 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:40.423625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:40.423628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:40.423680 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:40.426369 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:40.427202 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:40.427206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:40.427209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:40.427333 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:40.427791 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:40.427795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:40.427797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:40.428063 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:40.428213 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:40.428914 (+ 701us) server_negotiation.cc:299] Negotiation successful
0927 15:13:40.429099 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":282,"threads_started":1}
I20260927 15:13:40.430539 16458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:40.410198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32993 (local address 127.10.33.67:44725)
0927 15:13:40.411007 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:40.411053 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:40.413094 (+ 2041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:40.414337 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:40.414346 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:40.414950 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:40.415731 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:40.415756 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:40.417129 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:40.417134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:40.418975 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:40.418992 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:40.419975 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:40.419987 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:40.420274 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:40.420946 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:40.420971 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:40.423369 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:40.426530 (+ 3161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:40.426545 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:40.426562 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:40.427022 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:40.427465 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:40.427469 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:40.427471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:40.427684 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:40.428191 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:40.428197 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:40.428520 (+ 323us) client_negotiation.cc:769] Sending connection context
0927 15:13:40.428775 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 15:13:40.429226 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":334,"threads_started":1}
I20260927 15:13:40.431828 16455 heartbeater.cc:347] Connected to a master server at 127.10.33.126:32993
I20260927 15:13:40.432133 16455 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:40.432811 16455 heartbeater.cc:511] Master 127.10.33.126:32993 requested a full tablet report, sending...
I20260927 15:13:40.434238 15997 ts_manager.cc:194] Registered new tserver with Master: 91b7988fb71f4565b4b214412da2a113 (127.10.33.67:43171)
I20260927 15:13:40.435144 15997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:44725
I20260927 15:13:40.443843 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/test-admin.keytab.
Sep 27 15:13:40 dist-test-slave-f5h6 krb5kdc[15953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:40.471760 10373 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 15:13:40.476927 16463 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:13:40.474411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39436 (local address 127.10.33.65:35925)
0927 15:13:40.475272 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:40.475279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:40.475306 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:13:40.475369 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:40.475376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:40.475586 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:40.475736 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:40.476588 (+ 852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39436: BlockingRecv error: recv got EOF from 127.0.0.1:39436 (error 108)
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":472,"threads_started":1}
W20260927 15:13:40.477401 10373 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/krb5cc' of type FILE
Sep 27 15:13:40 dist-test-slave-f5h6 krb5kdc[15953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:40.478885 10373 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 15:13:40 dist-test-slave-f5h6 krb5kdc[15953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:40.480170 10373 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 15:13:40 dist-test-slave-f5h6 krb5kdc[15953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.65@KRBTEST.COM
I20260927 15:13:40.496551 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:40.481836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39452 (local address 127.10.33.65:35925)
0927 15:13:40.482048 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:40.482055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:40.482185 (+ 130us) server_negotiation.cc:407] Connection header received
0927 15:13:40.482368 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:40.482374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:40.482445 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:40.482535 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:40.483545 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:40.484530 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:40.485788 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:40.486427 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:40.486559 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:40.490608 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:40.490623 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:40.490640 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:40.490689 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:40.493473 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:40.494294 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:40.494305 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:40.494308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:40.494450 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:40.494848 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:40.494852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:40.494854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:40.495272 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:40.495574 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:40.495931 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 15:13:40.496214 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260927 15:13:40.502563 16463 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:13:40.500739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39454 (local address 127.10.33.65:35925)
0927 15:13:40.500923 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:40.500928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:40.501264 (+ 336us) server_negotiation.cc:407] Connection header received
0927 15:13:40.501414 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:40.501419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:40.501479 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:40.501556 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:40.502306 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39454: BlockingRecv error: recv got EOF from 127.0.0.1:39454 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 15:13:40.502964 10373 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/krb5cc)
Sep 27 15:13:40 dist-test-slave-f5h6 krb5kdc[15953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:40.504465 10373 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 15:13:40 dist-test-slave-f5h6 krb5kdc[15953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:40.505786 10373 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 15:13:40 dist-test-slave-f5h6 krb5kdc[15953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.65@KRBTEST.COM
I20260927 15:13:40.519892 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:40.507318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39466 (local address 127.10.33.65:35925)
0927 15:13:40.507537 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:40.507543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:40.507716 (+ 173us) server_negotiation.cc:407] Connection header received
0927 15:13:40.507830 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:40.507834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:40.507899 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:40.507988 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:40.508946 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:40.509694 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:40.511021 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:40.511446 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:40.511571 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:40.514909 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:40.514914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:40.514918 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:40.514964 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:40.517403 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:40.518100 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:40.518105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:40.518107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:40.518255 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:40.518661 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:40.518665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:40.518667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:40.518941 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:40.519178 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:40.519417 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 15:13:40.519594 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260927 15:13:40.524463 16463 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:13:40.522684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39480 (local address 127.10.33.65:35925)
0927 15:13:40.522822 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:40.522826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:40.523213 (+ 387us) server_negotiation.cc:407] Connection header received
0927 15:13:40.523356 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:40.523362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:40.523435 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:40.523516 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:40.524263 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39480: BlockingRecv error: recv got EOF from 127.0.0.1:39480 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 15:13:40.524804 10373 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521968392480-10373-0/krb5kdc/krb5cc' of type FILE
Sep 27 15:13:40 dist-test-slave-f5h6 krb5kdc[15953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:40.526364 10373 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 15:13:40 dist-test-slave-f5h6 krb5kdc[15953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:40.527760 10373 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 15:13:40 dist-test-slave-f5h6 krb5kdc[15953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.65@KRBTEST.COM
I20260927 15:13:40.542523 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:40.529145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39486 (local address 127.10.33.65:35925)
0927 15:13:40.529343 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:40.529349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:40.529590 (+ 241us) server_negotiation.cc:407] Connection header received
0927 15:13:40.529719 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:40.529722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:40.529781 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:40.529903 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:40.530795 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:40.531514 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:40.532683 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:40.533109 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:40.533202 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:40.536755 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:40.536761 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:40.536764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:40.536812 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:40.539502 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:40.540335 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:40.540341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:40.540343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:40.540519 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:40.540933 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:40.540937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:40.540939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:40.541298 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:40.541533 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:40.541887 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 15:13:40.542189 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:13:40.544147 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 16059
I20260927 15:13:40.556584 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:40.544878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39488 (local address 127.10.33.65:35925)
0927 15:13:40.545043 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:40.545051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:40.545310 (+ 259us) server_negotiation.cc:407] Connection header received
0927 15:13:40.545421 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:40.545424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:40.545484 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:40.545559 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:40.546475 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:40.547186 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:40.548596 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:40.549048 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:40.549148 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:40.551356 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:40.551361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:40.551364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:40.551399 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:40.553799 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:40.554633 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:40.554637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:40.554639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:40.554778 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:40.555130 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:40.555135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:40.555138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:40.555438 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:40.555671 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:40.555965 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 15:13:40.556179 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:13:40.557281 16182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:40.639796 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 16059
I20260927 15:13:40.653779 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 16193
Sep 27 15:13:40 dist-test-slave-f5h6 krb5kdc[15953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.66@KRBTEST.COM
I20260927 15:13:40.670751 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:40.654749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.10.33.66:39013)
0927 15:13:40.655265 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:40.655273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:40.655301 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:13:40.655371 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:40.655378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:40.655841 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:40.656109 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:40.656957 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:40.658357 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:40.659777 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:40.660355 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:40.660483 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:40.664555 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:40.664572 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:40.664587 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:40.664632 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:40.667674 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:40.668524 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:40.668532 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:40.668534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:40.668681 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:40.669046 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:40.669049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:40.669051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:40.669393 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:40.669638 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:40.670204 (+ 566us) server_negotiation.cc:299] Negotiation successful
0927 15:13:40.670437 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":256,"threads_started":1}
I20260927 15:13:40.671998 16316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:40.760561 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 16193
I20260927 15:13:40.772985 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 16327
Sep 27 15:13:40 dist-test-slave-f5h6 krb5kdc[15953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.67@KRBTEST.COM
I20260927 15:13:40.789093 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:40.774099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44854 (local address 127.10.33.67:43171)
0927 15:13:40.774766 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:40.774773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:40.774801 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:13:40.774858 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:40.774864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:40.775068 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:40.775200 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:40.776227 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:40.777232 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:40.778825 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:40.779482 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:40.779593 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:40.783470 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:40.783483 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:40.783498 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:40.783542 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:40.786356 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:40.787100 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:40.787108 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:40.787110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:40.787246 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:40.787607 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:40.787611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:40.787613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:40.787971 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:40.788273 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:40.788538 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 15:13:40.788800 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":365,"threads_started":1}
I20260927 15:13:40.790455 16450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:40.873580 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 16327
I20260927 15:13:40.886725 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 15966
I20260927 15:13:40.888146 16028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:40.992303 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 15966
2026-09-27T15:13:41Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3970 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:13:41 dist-test-slave-f5h6 krb5kdc[16483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:13:41 dist-test-slave-f5h6 krb5kdc[16483](info): set up 2 sockets
Sep 27 15:13:41 dist-test-slave-f5h6 krb5kdc[16483](info): commencing operation
krb5kdc: starting...
W20260927 15:13:42.034375 10373 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:13:42 dist-test-slave-f5h6 krb5kdc[16483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:13:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:13:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.126.keytab.
Entry for principal oryx/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.126.keytab.
I20260927 15:13:42.251307 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:32781
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.126.keytab
--principal=oryx/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:32781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:42.585247 16496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:42.585592 16496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:42.585737 16496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:42.593438 16496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:13:42.593585 16496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:42.593668 16496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:42.593744 16496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:13:42.593812 16496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:13:42.605393 16496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:32781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.126.keytab
--principal=oryx/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:32781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:42.606829 16496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:42.608207 16496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:42.619792 16501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:42.619827 16504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:42.619760 16502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:42.620615 16496 server_base.cc:1034] running on GCE node
I20260927 15:13:42.621527 16496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:42.623021 16496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:42.624269 16496 hybrid_clock.cc:648] HybridClock initialized: now 1790522022624198 us; error 53 us; skew 500 ppm
I20260927 15:13:42.624642 16496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:42 dist-test-slave-f5h6 krb5kdc[16483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522022, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:42.634473 16496 init.cc:377] Logged in from keytab as oryx/127.10.33.126@KRBTEST.COM (short username oryx)
I20260927 15:13:42.636158 16496 webserver.cc:460] Webserver started at http://127.10.33.126:34663/ using document root <none> and password file <none>
I20260927 15:13:42.636915 16496 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:42.637055 16496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:42.637355 16496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:42.640588 16496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "48af3088ca81428888ca24f47977bbe5"
format_stamp: "Formatted at 2026-09-27 15:13:42 on dist-test-slave-f5h6"
server_key: "625e7cf8e8b0a3e35568fbb337c1e11d"
server_key_iv: "eb035564a10c9a491caae9fccee42491"
server_key_version: "encryptionkey@0"
I20260927 15:13:42.641407 16496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "48af3088ca81428888ca24f47977bbe5"
format_stamp: "Formatted at 2026-09-27 15:13:42 on dist-test-slave-f5h6"
server_key: "625e7cf8e8b0a3e35568fbb337c1e11d"
server_key_iv: "eb035564a10c9a491caae9fccee42491"
server_key_version: "encryptionkey@0"
I20260927 15:13:42.647024 16496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 15:13:42.650959 16511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:42.652630 16496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:13:42.652814 16496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "48af3088ca81428888ca24f47977bbe5"
format_stamp: "Formatted at 2026-09-27 15:13:42 on dist-test-slave-f5h6"
server_key: "625e7cf8e8b0a3e35568fbb337c1e11d"
server_key_iv: "eb035564a10c9a491caae9fccee42491"
server_key_version: "encryptionkey@0"
I20260927 15:13:42.653023 16496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:42.680316 16496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:42.685158 16496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:42.685446 16496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:42.704562 16496 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:32781
I20260927 15:13:42.704623 16562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:32781 every 8 connection(s)
I20260927 15:13:42.706117 16496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:13:42.710122 16563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:42.712863 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 16496
I20260927 15:13:42.713066 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:13:42.713884 10373 external_mini_cluster.cc:1428] Setting key 487456d2c29a89c97f42d1991debcb37
I20260927 15:13:42.723500 16563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5: Bootstrap starting.
I20260927 15:13:42.727524 16563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:42.728910 16563 log.cc:824] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:13:42 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM
I20260927 15:13:42.732545 16563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5: No bootstrap required, opened a new log
I20260927 15:13:42.739717 16566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:42.716517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.10.33.126:32781)
0927 15:13:42.717324 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:42.717336 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:42.717379 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:13:42.719238 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:42.719265 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:42.719737 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:42.720132 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:42.722055 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:42.723748 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:42.725579 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:42.726414 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:42.726560 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:42.731560 (+ 5000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:42.731569 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:42.731585 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:42.731631 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:42.735013 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:42.736010 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:42.736017 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:42.736027 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:42.736221 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:42.736657 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:42.736666 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:42.736668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:42.737284 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:42.737578 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:42.738109 (+ 531us) server_negotiation.cc:299] Negotiation successful
0927 15:13:42.738598 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":368,"threads_started":1}
I20260927 15:13:42.741580 16563 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } }
I20260927 15:13:42.741969 16563 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:42.742057 16563 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48af3088ca81428888ca24f47977bbe5, State: Initialized, Role: FOLLOWER
I20260927 15:13:42.742604 16563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } }
I20260927 15:13:42.742818 16563 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:13:42.742933 16563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:13:42.743069 16563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:42.745193 16563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } }
I20260927 15:13:42.745669 16563 leader_election.cc:302] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48af3088ca81428888ca24f47977bbe5; no voters:
I20260927 15:13:42.746266 16563 leader_election.cc:288] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:13:42.746377 16568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:42.747037 16568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [term 1 LEADER]: Becoming Leader. State: Replica: 48af3088ca81428888ca24f47977bbe5, State: Running, Role: LEADER
I20260927 15:13:42.747677 16568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } }
I20260927 15:13:42.748373 16563 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:13:42.750641 16570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48af3088ca81428888ca24f47977bbe5. Latest consensus state: current_term: 1 leader_uuid: "48af3088ca81428888ca24f47977bbe5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } }
I20260927 15:13:42.751035 16570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:42.750799 16569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48af3088ca81428888ca24f47977bbe5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } }
I20260927 15:13:42.751194 16569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:42.752975 16575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:13:42.759073 16575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:13:42.768529 16575 catalog_manager.cc:1347] Generated new cluster ID: 4d4d860505ed4bc6aeb9e0803aa6f40d
I20260927 15:13:42.768668 16575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:13:42.781999 16575 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:13:42.783376 16575 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:13:42.789947 16575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5: Generated new TSK 0
I20260927 15:13:42.790987 16575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.33.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.33.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.33.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.65.keytab.
Entry for principal oryx/127.10.33.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.65.keytab.
I20260927 15:13:42.858621 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:32781
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.65.keytab
--principal=oryx/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:43.222730 16589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:43.223044 16589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:43.223137 16589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:43.230628 16589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:43.230751 16589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:43.230892 16589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:13:43.242012 16589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.65.keytab
--principal=oryx/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:32781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:43.243356 16589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:43.244755 16589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:43.258105 16595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:43.258255 16594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:43.258229 16597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:43.259255 16589 server_base.cc:1034] running on GCE node
I20260927 15:13:43.259872 16589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:43.260883 16589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:43.262077 16589 hybrid_clock.cc:648] HybridClock initialized: now 1790522023262029 us; error 40 us; skew 500 ppm
I20260927 15:13:43.262403 16589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:43 dist-test-slave-f5h6 krb5kdc[16483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522023, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:43.265713 16589 init.cc:377] Logged in from keytab as oryx/127.10.33.65@KRBTEST.COM (short username oryx)
I20260927 15:13:43.267357 16589 webserver.cc:460] Webserver started at http://127.10.33.65:46519/ using document root <none> and password file <none>
I20260927 15:13:43.268038 16589 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:43.268160 16589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:43.268435 16589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:43.271415 16589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3"
format_stamp: "Formatted at 2026-09-27 15:13:43 on dist-test-slave-f5h6"
server_key: "8bcdaf4efcfbdd3e090fbf0f7ccc5f57"
server_key_iv: "a6c4ad9e88d1c000e414b1c8e1ea9d02"
server_key_version: "encryptionkey@0"
I20260927 15:13:43.272168 16589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3"
format_stamp: "Formatted at 2026-09-27 15:13:43 on dist-test-slave-f5h6"
server_key: "8bcdaf4efcfbdd3e090fbf0f7ccc5f57"
server_key_iv: "a6c4ad9e88d1c000e414b1c8e1ea9d02"
server_key_version: "encryptionkey@0"
I20260927 15:13:43.277245 16589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:13:43.280869 16604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:43.282519 16589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 15:13:43.282783 16589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3"
format_stamp: "Formatted at 2026-09-27 15:13:43 on dist-test-slave-f5h6"
server_key: "8bcdaf4efcfbdd3e090fbf0f7ccc5f57"
server_key_iv: "a6c4ad9e88d1c000e414b1c8e1ea9d02"
server_key_version: "encryptionkey@0"
I20260927 15:13:43.283022 16589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:43.298707 16589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:43.303202 16589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:43.303474 16589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:43.304440 16589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:43.305774 16589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:43.305850 16589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:43.305994 16589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:43.306049 16589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:43.336426 16589 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:41861
I20260927 15:13:43.336472 16716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:41861 every 8 connection(s)
I20260927 15:13:43.337692 16589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:13:43.342814 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 16589
I20260927 15:13:43.343009 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:13:43.343617 10373 external_mini_cluster.cc:1428] Setting key a1e78564d6d1f7142325952556e6757d
Sep 27 15:13:43 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522023, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM
I20260927 15:13:43.358800 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:43.340266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:59243 (local address 127.10.33.126:32781)
0927 15:13:43.340736 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:43.340741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:43.341619 (+ 878us) server_negotiation.cc:407] Connection header received
0927 15:13:43.343500 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:43.343507 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:43.343588 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:43.343682 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:43.345582 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:43.346720 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:43.348373 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:43.348781 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:43.348918 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:43.352469 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:43.352474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:43.352477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:43.352510 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:43.355639 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:43.356461 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:43.356468 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:43.356470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:43.356592 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:43.357023 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:43.357027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:43.357029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:43.357308 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:43.357439 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:43.358226 (+ 787us) server_negotiation.cc:299] Negotiation successful
0927 15:13:43.358447 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":230,"threads_started":1}
I20260927 15:13:43.359799 16720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:43.340701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.10.33.65:59243)
0927 15:13:43.341423 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:43.341469 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:43.343226 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:43.343930 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:43.343939 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:43.344554 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:43.345368 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:43.345396 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:43.346890 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:43.346894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:43.348193 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:43.348206 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:43.349024 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:43.349032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:43.349284 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:43.350025 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:43.350049 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:43.352220 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:43.355830 (+ 3610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:43.355838 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:43.355854 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:43.356317 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:43.356709 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:43.356713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:43.356715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:43.356913 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:43.357433 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:43.357442 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:43.357817 (+ 375us) client_negotiation.cc:769] Sending connection context
0927 15:13:43.358108 (+ 291us) client_negotiation.cc:240] Negotiation successful
0927 15:13:43.358671 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":276,"threads_started":1}
I20260927 15:13:43.361325 16717 heartbeater.cc:347] Connected to a master server at 127.10.33.126:32781
I20260927 15:13:43.361701 16717 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.10.33.66@KRBTEST.COM; defaulting to no policy
I20260927 15:13:43.362690 16717 heartbeater.cc:511] Master 127.10.33.126:32781 requested a full tablet report, sending...
I20260927 15:13:43.365085 16528 ts_manager.cc:194] Registered new tserver with Master: 48e1d19f3efd444fa8f0ba4fea3b7da3 (127.10.33.65:41861)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.33.66@KRBTEST.COM" created.
I20260927 15:13:43.366648 16528 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:59243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.33.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.66.keytab.
Entry for principal oryx/127.10.33.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.66.keytab.
I20260927 15:13:43.389727 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:32781
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.66.keytab
--principal=oryx/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:43.714188 16723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:43.714509 16723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:43.714653 16723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:43.722580 16723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:43.722751 16723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:43.722910 16723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:13:43.733211 16723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.66.keytab
--principal=oryx/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:32781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:43.734632 16723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:43.735985 16723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:43.748812 16728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:43.748812 16729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:43.749068 16731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:43.749948 16723 server_base.cc:1034] running on GCE node
I20260927 15:13:43.750655 16723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:43.751654 16723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:43.752888 16723 hybrid_clock.cc:648] HybridClock initialized: now 1790522023752816 us; error 57 us; skew 500 ppm
I20260927 15:13:43.753222 16723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:43 dist-test-slave-f5h6 krb5kdc[16483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522023, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:43.756337 16723 init.cc:377] Logged in from keytab as oryx/127.10.33.66@KRBTEST.COM (short username oryx)
I20260927 15:13:43.757833 16723 webserver.cc:460] Webserver started at http://127.10.33.66:45045/ using document root <none> and password file <none>
I20260927 15:13:43.758565 16723 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:43.758694 16723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:43.758952 16723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:43.762007 16723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "49233ab4d77a4c58b34c9f40d42ff227"
format_stamp: "Formatted at 2026-09-27 15:13:43 on dist-test-slave-f5h6"
server_key: "89e72945678b9ce155eaa375576b6e21"
server_key_iv: "91112c6990af562f48a4cb5e9c462db1"
server_key_version: "encryptionkey@0"
I20260927 15:13:43.762799 16723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "49233ab4d77a4c58b34c9f40d42ff227"
format_stamp: "Formatted at 2026-09-27 15:13:43 on dist-test-slave-f5h6"
server_key: "89e72945678b9ce155eaa375576b6e21"
server_key_iv: "91112c6990af562f48a4cb5e9c462db1"
server_key_version: "encryptionkey@0"
I20260927 15:13:43.768142 16723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:13:43.771821 16738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:43.773723 16723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:13:43.773958 16723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "49233ab4d77a4c58b34c9f40d42ff227"
format_stamp: "Formatted at 2026-09-27 15:13:43 on dist-test-slave-f5h6"
server_key: "89e72945678b9ce155eaa375576b6e21"
server_key_iv: "91112c6990af562f48a4cb5e9c462db1"
server_key_version: "encryptionkey@0"
I20260927 15:13:43.774166 16723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:43.800436 16723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:43.805719 16723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:43.806038 16723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:43.807011 16723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:43.808378 16723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:43.808454 16723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:43.808569 16723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:43.808620 16723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:43.839335 16723 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:37121
I20260927 15:13:43.839376 16850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:37121 every 8 connection(s)
I20260927 15:13:43.840543 16723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:13:43.850238 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 16723
I20260927 15:13:43.850440 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:13:43.851044 10373 external_mini_cluster.cc:1428] Setting key a3cd036f4da1b6cb7fc0895f7d41440b
Sep 27 15:13:43 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522023, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM
I20260927 15:13:43.862452 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:43.843123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:50111 (local address 127.10.33.126:32781)
0927 15:13:43.843345 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:43.843351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:43.844491 (+ 1140us) server_negotiation.cc:407] Connection header received
0927 15:13:43.846410 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:43.846415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:43.846499 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:43.846702 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:43.848332 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:43.849094 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:43.851601 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:43.852157 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:43.852282 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:43.856394 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:43.856400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:43.856403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:43.856437 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:43.858996 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:43.859876 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:43.859881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:43.859883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:43.860011 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:43.860435 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:43.860438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:43.860440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:43.860817 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:43.860973 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:43.861817 (+ 844us) server_negotiation.cc:299] Negotiation successful
0927 15:13:43.862106 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 15:13:43.863480 16853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:43.843535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.10.33.66:50111)
0927 15:13:43.844317 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:43.844358 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:43.846085 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:43.846922 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:43.846932 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:43.847556 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:43.848153 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:43.848172 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:43.849292 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:43.849297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:43.851376 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:43.851398 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:43.852410 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:43.852417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:43.852667 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:43.853588 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:43.853613 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:43.856127 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:43.859163 (+ 3036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:43.859173 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:43.859193 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:43.859737 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:43.860119 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:43.860123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:43.860125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:43.860319 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:43.860973 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:43.860983 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:43.861341 (+ 358us) client_negotiation.cc:769] Sending connection context
0927 15:13:43.861587 (+ 246us) client_negotiation.cc:240] Negotiation successful
0927 15:13:43.862289 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":310,"threads_started":1}
I20260927 15:13:43.865027 16851 heartbeater.cc:347] Connected to a master server at 127.10.33.126:32781
I20260927 15:13:43.865368 16851 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:43.866117 16851 heartbeater.cc:511] Master 127.10.33.126:32781 requested a full tablet report, sending...
I20260927 15:13:43.867805 16528 ts_manager.cc:194] Registered new tserver with Master: 49233ab4d77a4c58b34c9f40d42ff227 (127.10.33.66:37121)
I20260927 15:13:43.868942 16528 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:50111
WARNING: no policy specified for oryx/127.10.33.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.33.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.33.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.67.keytab.
Entry for principal oryx/127.10.33.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.67.keytab.
I20260927 15:13:43.898883 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:32781
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.67.keytab
--principal=oryx/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:44.227545 16856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:44.227839 16856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:44.227955 16856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:44.235455 16856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:44.235586 16856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:44.235740 16856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:13:44.245687 16856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.67.keytab
--principal=oryx/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:32781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:44.247051 16856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:44.248412 16856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:44.261456 16861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:44.261489 16862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:44.261718 16864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:44.262262 16856 server_base.cc:1034] running on GCE node
I20260927 15:13:44.262848 16856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:44.263769 16856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:44.264971 16856 hybrid_clock.cc:648] HybridClock initialized: now 1790522024264921 us; error 47 us; skew 500 ppm
I20260927 15:13:44.265270 16856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:44 dist-test-slave-f5h6 krb5kdc[16483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522024, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:44.268875 16856 init.cc:377] Logged in from keytab as oryx/127.10.33.67@KRBTEST.COM (short username oryx)
I20260927 15:13:44.270560 16856 webserver.cc:460] Webserver started at http://127.10.33.67:38807/ using document root <none> and password file <none>
I20260927 15:13:44.271214 16856 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:44.271313 16856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:44.271548 16856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:44.274483 16856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "8f1dea582f4446818f93998cdd54100c"
format_stamp: "Formatted at 2026-09-27 15:13:44 on dist-test-slave-f5h6"
server_key: "a4a07f57a542449cbaa152dbe55a5f72"
server_key_iv: "5324a59cdc599757084895fa10fc4b1b"
server_key_version: "encryptionkey@0"
I20260927 15:13:44.275213 16856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "8f1dea582f4446818f93998cdd54100c"
format_stamp: "Formatted at 2026-09-27 15:13:44 on dist-test-slave-f5h6"
server_key: "a4a07f57a542449cbaa152dbe55a5f72"
server_key_iv: "5324a59cdc599757084895fa10fc4b1b"
server_key_version: "encryptionkey@0"
I20260927 15:13:44.280601 16856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:13:44.284258 16871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:44.285782 16856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:13:44.285998 16856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "8f1dea582f4446818f93998cdd54100c"
format_stamp: "Formatted at 2026-09-27 15:13:44 on dist-test-slave-f5h6"
server_key: "a4a07f57a542449cbaa152dbe55a5f72"
server_key_iv: "5324a59cdc599757084895fa10fc4b1b"
server_key_version: "encryptionkey@0"
I20260927 15:13:44.286201 16856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:44.313755 16856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:44.318758 16856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:44.319033 16856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:44.320221 16856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:44.321692 16856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:44.321772 16856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:44.321895 16856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:44.321956 16856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 15:13:44.353881 16856 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:34725
I20260927 15:13:44.353925 16983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:34725 every 8 connection(s)
I20260927 15:13:44.355224 16856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:13:44.359748 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 16856
I20260927 15:13:44.359892 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:13:44.360320 10373 external_mini_cluster.cc:1428] Setting key 8e8a557d8f686eb6908b78f1cf707558
Sep 27 15:13:44 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522024, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM
I20260927 15:13:44.369829 16717 heartbeater.cc:503] Master 127.10.33.126:32781 was elected leader, sending a full tablet report...
I20260927 15:13:44.376251 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:44.357716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:50531 (local address 127.10.33.126:32781)
0927 15:13:44.357946 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:44.357952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:44.359154 (+ 1202us) server_negotiation.cc:407] Connection header received
0927 15:13:44.360982 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:44.360988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:44.361094 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:44.361188 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:44.362982 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.363807 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.365366 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.365796 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.365940 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:44.369271 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:44.369276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:44.369279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:44.369311 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:44.372867 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:44.373706 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:44.373710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:44.373713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:44.373842 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:44.374372 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:44.374377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:44.374380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:44.374673 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:44.374815 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:44.375666 (+ 851us) server_negotiation.cc:299] Negotiation successful
0927 15:13:44.375897 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:13:44.377269 16986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:44.358256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.10.33.67:50531)
0927 15:13:44.358986 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:44.359031 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:44.360703 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:44.361379 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:44.361389 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:44.361995 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:44.362783 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:44.362810 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.363954 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.363959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:44.365218 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:44.365230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.366002 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.366007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:44.366215 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:44.366848 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:44.366871 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:44.369053 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:44.373031 (+ 3978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:44.373042 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:44.373061 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:44.373566 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:44.374004 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:44.374009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:44.374013 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:44.374269 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:44.374804 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:44.374812 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:44.375270 (+ 458us) client_negotiation.cc:769] Sending connection context
0927 15:13:44.375530 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 15:13:44.376112 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":293,"threads_started":1}
I20260927 15:13:44.378573 16984 heartbeater.cc:347] Connected to a master server at 127.10.33.126:32781
I20260927 15:13:44.378891 16984 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:44.379614 16984 heartbeater.cc:511] Master 127.10.33.126:32781 requested a full tablet report, sending...
I20260927 15:13:44.380923 16528 ts_manager.cc:194] Registered new tserver with Master: 8f1dea582f4446818f93998cdd54100c (127.10.33.67:34725)
I20260927 15:13:44.381726 16528 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:50531
I20260927 15:13:44.388048 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.33.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.33.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.33.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.125.keytab.
Entry for principal oryx/127.10.33.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.125.keytab.
I20260927 15:13:44.434834 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.125:37211
--webserver_interface=127.10.33.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.125.keytab
--principal=oryx/127.10.33.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:32781,127.10.33.125:37211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:44.756613 16989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:44.756968 16989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:44.757097 16989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:44.764657 16989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:13:44.764807 16989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:44.764868 16989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:44.764948 16989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:13:44.764994 16989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:13:44.775413 16989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:32781,127.10.33.125:37211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.125.keytab
--principal=oryx/127.10.33.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.125:37211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.33.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
W20260927 15:13:44.775825 16989 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.10.33.126:32781,127.10.33.125:37211'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 15:13:44.776901 16989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:44.778339 16989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:44.790055 16997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:44.790117 16995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:44.790252 16994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:44.791306 16989 server_base.cc:1034] running on GCE node
I20260927 15:13:44.792079 16989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:44.793538 16989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:44.794847 16989 hybrid_clock.cc:648] HybridClock initialized: now 1790522024794759 us; error 61 us; skew 500 ppm
I20260927 15:13:44.795184 16989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:44 dist-test-slave-f5h6 krb5kdc[16483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522024, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:44.798658 16989 init.cc:377] Logged in from keytab as oryx/127.10.33.125@KRBTEST.COM (short username oryx)
I20260927 15:13:44.800226 16989 webserver.cc:460] Webserver started at http://127.10.33.125:43801/ using document root <none> and password file <none>
I20260927 15:13:44.800908 16989 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:44.800997 16989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:44.801262 16989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:44.804297 16989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/data/instance:
uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2"
format_stamp: "Formatted at 2026-09-27 15:13:44 on dist-test-slave-f5h6"
server_key: "42418302107e8e1d33a015b3512615fc"
server_key_iv: "32680da78bb6432a73d6042710a5b7ba"
server_key_version: "encryptionkey@0"
I20260927 15:13:44.805087 16989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/wal/instance:
uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2"
format_stamp: "Formatted at 2026-09-27 15:13:44 on dist-test-slave-f5h6"
server_key: "42418302107e8e1d33a015b3512615fc"
server_key_iv: "32680da78bb6432a73d6042710a5b7ba"
server_key_version: "encryptionkey@0"
I20260927 15:13:44.810509 16989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 15:13:44.814247 17004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:44.815830 16989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:13:44.816056 16989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/wal
uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2"
format_stamp: "Formatted at 2026-09-27 15:13:44 on dist-test-slave-f5h6"
server_key: "42418302107e8e1d33a015b3512615fc"
server_key_iv: "32680da78bb6432a73d6042710a5b7ba"
server_key_version: "encryptionkey@0"
I20260927 15:13:44.816275 16989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:44.831133 16989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:44.835378 16989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:44.835593 16989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:44.853310 16989 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.125:37211
I20260927 15:13:44.853359 17055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.125:37211 every 8 connection(s)
I20260927 15:13:44.854774 16989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/data/info.pb
I20260927 15:13:44.858763 17056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:44.865113 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 16989
I20260927 15:13:44.865252 17056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:13:44.871809 16851 heartbeater.cc:503] Master 127.10.33.126:32781 was elected leader, sending a full tablet report...
Sep 27 15:13:44 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522024, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.125@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM
Sep 27 15:13:44 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.33.125@KRBTEST.COM
I20260927 15:13:44.888866 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:44.868347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54390 (local address 127.10.33.126:32781)
0927 15:13:44.868530 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:44.868537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:44.869379 (+ 842us) server_negotiation.cc:407] Connection header received
0927 15:13:44.870494 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:44.870501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:44.870578 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:44.870669 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:44.872487 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.873338 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.875475 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.876000 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.876123 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:44.880224 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:44.880231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:44.880357 (+ 126us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:44.880408 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:44.885234 (+ 4826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:44.886201 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:44.886207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:44.886210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:44.886377 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:44.886863 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:44.886868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:44.886871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:44.887324 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:44.887513 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:44.888240 (+ 727us) server_negotiation.cc:299] Negotiation successful
0927 15:13:44.888487 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:13:44.890223 17060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:44.868299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.0.0.1:54390)
0927 15:13:44.869124 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:44.869167 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:44.870217 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:44.870749 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:44.870756 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:44.871668 (+ 912us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:44.872195 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:44.872229 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.873510 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.873516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:44.875298 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:44.875309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.876264 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.876272 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:44.876643 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:44.877398 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:44.877433 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:44.879889 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:44.885377 (+ 5488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:44.885391 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:44.885406 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:44.886037 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:44.886493 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:44.886497 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:44.886500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:44.886722 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:44.887536 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:44.887545 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:44.887802 (+ 257us) client_negotiation.cc:769] Sending connection context
0927 15:13:44.888102 (+ 300us) client_negotiation.cc:240] Negotiation successful
0927 15:13:44.888676 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":46,"thread_start_us":403,"threads_started":1}
I20260927 15:13:44.890251 17059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:44.867406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56010 (local address 127.10.33.125:37211)
0927 15:13:44.868100 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:44.868114 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:44.868166 (+ 52us) server_negotiation.cc:407] Connection header received
0927 15:13:44.870190 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:44.870215 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:44.870490 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:44.870708 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:44.872537 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.874330 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.875764 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.876587 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.876715 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:44.880991 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:44.881000 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:44.881016 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:44.881048 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:44.885000 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:44.886006 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:44.886013 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:44.886016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:44.886247 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:44.886724 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:44.886728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:44.886731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:44.887535 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:44.887835 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:44.888443 (+ 608us) server_negotiation.cc:299] Negotiation successful
0927 15:13:44.888850 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":233,"server-negotiator.queue_time_us":501,"thread_start_us":308,"threads_started":1}
I20260927 15:13:44.893079 17056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:13:44 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522024, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.125@KRBTEST.COM for oryx/127.10.33.125@KRBTEST.COM
I20260927 15:13:44.905404 17060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:44.893776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:37211 (local address 127.0.0.1:56016)
0927 15:13:44.894005 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:44.894022 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:44.894113 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:44.894409 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:44.894413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:44.894769 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:44.894904 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:44.894912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.895963 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.895968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:44.896858 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:44.896870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.897586 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.897591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:44.897763 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:44.898370 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:44.898387 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:44.900324 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:44.903212 (+ 2888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:44.903216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:44.903219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:44.903586 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:44.903938 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:44.903941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:44.903943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:44.904074 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:44.904629 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:44.904633 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:44.904782 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 15:13:44.904908 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 15:13:44.905151 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260927 15:13:44.905575 17059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:44.893982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.10.33.125:37211)
0927 15:13:44.894154 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:44.894159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:44.894174 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:13:44.894215 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:44.894218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:44.894270 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:44.894346 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:44.895042 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.895816 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.897004 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.897421 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.897531 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:44.900492 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:44.900497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:44.900500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:44.900529 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:44.903021 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:44.903711 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:44.903715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:44.903717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:44.903840 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:44.904210 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:44.904215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:44.904218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:44.904502 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:44.904613 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:44.905000 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 15:13:44.905262 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":43}
I20260927 15:13:44.910354 17056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2: Bootstrap starting.
I20260927 15:13:44.913741 17056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:44.914897 17056 log.cc:824] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:44.917739 17056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2: No bootstrap required, opened a new log
I20260927 15:13:44.925289 17056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } }
I20260927 15:13:44.925668 17056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:44.925765 17056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bf082f24f3f46be9dcb1fddb0ff06c2, State: Initialized, Role: FOLLOWER
I20260927 15:13:44.926383 17056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } }
I20260927 15:13:44.927258 17062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } } }
I20260927 15:13:44.927601 17062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:13:44.928263 17056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 15:13:44.937134 17073 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:13:44.937350 17073 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:13:44.939888 16989 master_runner.cc:428] Detected that this master 4bf082f24f3f46be9dcb1fddb0ff06c2 is joining an existing cluster
I20260927 15:13:44.940001 16989 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.33.125:37211
I20260927 15:13:44.959714 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:44.944183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54400 (local address 127.10.33.126:32781)
0927 15:13:44.944367 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:44.944375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:44.945086 (+ 711us) server_negotiation.cc:407] Connection header received
0927 15:13:44.945342 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:44.945348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:44.945427 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:44.945496 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:44.946998 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.948014 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.950051 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.950629 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.950736 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:44.953191 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:44.953198 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:44.953202 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:44.953241 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:44.956498 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:44.957320 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:44.957325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:44.957328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:44.957473 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:44.958023 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:44.958028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:44.958030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:44.958370 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:44.958479 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:44.959125 (+ 646us) server_negotiation.cc:299] Negotiation successful
0927 15:13:44.959373 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:13:44.960938 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:44.944237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.0.0.1:54400)
0927 15:13:44.944984 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:44.945007 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:44.945198 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:44.945714 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:44.945719 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:44.946375 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:44.946824 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:44.946837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.948201 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.948205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:44.949844 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:44.949883 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.951034 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.951041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:44.951196 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:44.951872 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:44.951897 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:44.952966 (+ 1069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:44.956725 (+ 3759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:44.956730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:44.956733 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:44.957192 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:44.957631 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:44.957635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:44.957638 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:44.957907 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:44.958579 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:44.958584 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:44.958810 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 15:13:44.960108 (+ 1298us) client_negotiation.cc:240] Negotiation successful
0927 15:13:44.960454 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":406,"threads_started":1}
WARNING: no policy specified for oryx/127.10.33.124@KRBTEST.COM; defaulting to no policy
I20260927 15:13:44.970932 17059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:44.945201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56030 (local address 127.10.33.125:37211)
0927 15:13:44.945447 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:44.945452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:44.955251 (+ 9799us) server_negotiation.cc:407] Connection header received
0927 15:13:44.955396 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:44.955401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:44.955464 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:44.955555 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:44.956967 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.958123 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.960191 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.960957 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.961055 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:44.963619 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:44.963625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:44.963629 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:44.963658 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:44.967659 (+ 4001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:44.968594 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:44.968600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:44.968607 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:44.968778 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:44.969194 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:44.969199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:44.969202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:44.969625 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:44.969760 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:44.970283 (+ 523us) server_negotiation.cc:299] Negotiation successful
0927 15:13:44.970574 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260927 15:13:44.971850 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:44.945326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:37211 (local address 127.0.0.1:56030)
0927 15:13:44.955143 (+ 9817us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:44.955159 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:44.955291 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:44.955708 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:44.955713 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:44.956437 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:44.956803 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:44.956817 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.958417 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.958423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:44.960016 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:44.960036 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.961381 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.961388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:44.961558 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:44.962310 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:44.962330 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:44.963438 (+ 1108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:44.967866 (+ 4428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:44.967871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:44.967875 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:44.968458 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:44.968923 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:44.968928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:44.968932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:44.969083 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:44.969851 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:44.969879 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:44.970035 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 15:13:44.971181 (+ 1146us) client_negotiation.cc:240] Negotiation successful
0927 15:13:44.971415 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9698,"thread_start_us":9521,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.33.124@KRBTEST.COM" created.
Sep 27 15:13:44 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522022, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.126@KRBTEST.COM for oryx/127.10.33.125@KRBTEST.COM
I20260927 15:13:44.996433 17082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:44.968099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:37211 (local address 127.0.0.1:56042)
0927 15:13:44.978117 (+ 10018us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:44.978185 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:44.978792 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:44.979547 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:44.979555 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:44.980199 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:44.980404 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:44.980413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.981836 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.981843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:44.983472 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:44.983490 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.984723 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.984730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:44.984946 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:44.985713 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:44.985740 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:44.988725 (+ 2985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:44.993047 (+ 4322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:44.993061 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:44.993068 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:44.993769 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:44.994334 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:44.994340 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:44.994343 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:44.994519 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:44.995222 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:44.995229 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:44.995408 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 15:13:44.995701 (+ 293us) client_negotiation.cc:240] Negotiation successful
0927 15:13:44.996017 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9774,"thread_start_us":9586,"threads_started":1}
I20260927 15:13:44.996620 17059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:44.968457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56042 (local address 127.10.33.125:37211)
0927 15:13:44.971146 (+ 2689us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:44.971151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:44.978288 (+ 7137us) server_negotiation.cc:407] Connection header received
0927 15:13:44.979047 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:44.979054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:44.979173 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:44.979246 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:44.980580 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.981679 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.983646 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:44.984330 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:44.984452 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:44.989003 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:44.989011 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:44.989016 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:44.989060 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:44.992830 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:44.994000 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:44.994005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:44.994008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:44.994161 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:44.994678 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:44.994685 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:44.994689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:44.995068 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:44.995226 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:44.995897 (+ 671us) server_negotiation.cc:299] Negotiation successful
0927 15:13:44.996166 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2591,"thread_start_us":9976,"threads_started":1}
I20260927 15:13:44.998510 16528 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.33.125:37211: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2"
member_type: NON_VOTER
last_known_addr {
host: "127.10.33.125"
port: 37211
}
attrs {
promote: true
}
}
dest_uuid: "48af3088ca81428888ca24f47977bbe5"
cas_config_opid_index: -1
I20260927 15:13:44.999709 16528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: NON_VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.33.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.124.keytab.
Entry for principal oryx/127.10.33.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.124.keytab.
I20260927 15:13:45.002645 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.124:45567
--webserver_interface=127.10.33.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.124.keytab
--principal=oryx/127.10.33.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:32781,127.10.33.125:37211,127.10.33.124:45567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:13:45.009387 17087 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4bf082f24f3f46be9dcb1fddb0ff06c2 (127.10.33.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: NON_VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: true } } }
I20260927 15:13:45.010566 17031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:45.013321 17087 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.33.125:37211
I20260927 15:13:45.013764 17087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "48af3088ca81428888ca24f47977bbe5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: NON_VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: true } } }
I20260927 15:13:45.014186 16989 master.cc:537] Master@127.10.33.125:37211 shutting down...
I20260927 15:13:45.014272 17031 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 1 FOLLOWER]: Refusing update from remote peer 48af3088ca81428888ca24f47977bbe5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 15:13:45.014037 17087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:45.018301 17088 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: NON_VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:13:45.022205 16512 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 15:13:45.026057 16989 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:45.026518 16989 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:45.026726 16989 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2: stopping tablet replica
I20260927 15:13:45.035633 17082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.035052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:37211 (local address 127.0.0.1:56056)
0927 15:13:45.035257 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:45.035402 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:37211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 15:13:45.035893 16512 consensus_peers.cc:487] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 -> Peer 4bf082f24f3f46be9dcb1fddb0ff06c2 (127.10.33.125:37211): Couldn't send request to peer 4bf082f24f3f46be9dcb1fddb0ff06c2. Status: Network error: Client connection negotiation failed: client connection to 127.10.33.125:37211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 15:13:45.037335 16989 master.cc:552] Master@127.10.33.125:37211 shutdown complete.
I20260927 15:13:45.037416 16989 master_runner.cc:305] Clearing existing system tablet
I20260927 15:13:45.037779 16989 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:45.037911 16989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:45.040210 16989 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260927 15:13:45.041687 17092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:45.041985 16989 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:45.042248 16989 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 15:13:45.042335 16989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/wal
uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2"
format_stamp: "Formatted at 2026-09-27 15:13:44 on dist-test-slave-f5h6"
server_key: "42418302107e8e1d33a015b3512615fc"
server_key_iv: "32680da78bb6432a73d6042710a5b7ba"
server_key_version: "encryptionkey@0"
I20260927 15:13:45.045782 16989 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:45.050868 16989 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 15:13:45.051060 16989 log.cc:1196] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 15:13:45.051414 16989 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2: Deleting consensus metadata
I20260927 15:13:45.051638 16989 master_runner.cc:315] Copying system tablet from 127.10.33.126:32781
I20260927 15:13:45.062294 16989 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2: tablet copy: Beginning tablet copy session from remote peer at address 127.10.33.126:32781
I20260927 15:13:45.077131 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.063090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.10.33.126:32781)
0927 15:13:45.063275 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:45.063280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:45.066392 (+ 3112us) server_negotiation.cc:407] Connection header received
0927 15:13:45.066592 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:45.066598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:45.066681 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:45.066764 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:45.067673 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.068384 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.069588 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.070115 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.070291 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:45.072047 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:45.072052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:45.072055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:45.072087 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:45.074627 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:45.075272 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:45.075277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:45.075280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:45.075404 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:45.075770 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:45.075775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:45.075779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:45.076054 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:45.076185 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:45.076574 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 15:13:45.076789 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:13:45.077227 17098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.063086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.0.0.1:54404)
0927 15:13:45.066269 (+ 3183us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:45.066294 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:45.066455 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:45.066817 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:45.066822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:45.067277 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:45.067521 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:45.067533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.068544 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.068549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:45.069433 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:45.069446 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.070279 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.070286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:45.070471 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:45.071111 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:45.071128 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:45.071847 (+ 719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:45.074772 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:45.074778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:45.074782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:45.075155 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:45.075512 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:45.075517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:45.075521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:45.075662 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:45.076166 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:45.076171 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:45.076331 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 15:13:45.076461 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 15:13:45.076792 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3048,"thread_start_us":2941,"threads_started":1}
I20260927 15:13:45.078263 16548 tablet_copy_service.cc:140] P 48af3088ca81428888ca24f47977bbe5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4bf082f24f3f46be9dcb1fddb0ff06c2 ({username='oryx', principal='oryx/127.10.33.125@KRBTEST.COM'} at 127.0.0.1:54404)
I20260927 15:13:45.078504 16548 tablet_copy_service.cc:161] P 48af3088ca81428888ca24f47977bbe5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4bf082f24f3f46be9dcb1fddb0ff06c2 at {username='oryx', principal='oryx/127.10.33.125@KRBTEST.COM'} at 127.0.0.1:54404: session id = 4bf082f24f3f46be9dcb1fddb0ff06c2-00000000000000000000000000000000
I20260927 15:13:45.081583 16548 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 15:13:45.083433 16989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 15:13:45.088174 16989 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2: tablet copy: Starting download of 0 data blocks...
I20260927 15:13:45.088449 16989 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2: tablet copy: Starting download of 1 WAL segments...
I20260927 15:13:45.102269 16989 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 15:13:45.105984 16548 tablet_copy_service.cc:342] P 48af3088ca81428888ca24f47977bbe5: Request end of tablet copy session 4bf082f24f3f46be9dcb1fddb0ff06c2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.33.125@KRBTEST.COM'} at 127.0.0.1:54404
I20260927 15:13:45.106187 16548 tablet_copy_service.cc:433] P 48af3088ca81428888ca24f47977bbe5: ending tablet copy session 4bf082f24f3f46be9dcb1fddb0ff06c2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4bf082f24f3f46be9dcb1fddb0ff06c2
I20260927 15:13:45.116614 16989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:45.119280 17099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:45.123561 17100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:45.134089 16989 server_base.cc:1034] running on GCE node
W20260927 15:13:45.134320 17102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:45.134779 16989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:45.135394 16989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:45.136606 16989 hybrid_clock.cc:648] HybridClock initialized: now 1790522025136546 us; error 35 us; skew 500 ppm
I20260927 15:13:45.136832 16989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:45 dist-test-slave-f5h6 krb5kdc[16483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522025, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:45.138641 16989 init.cc:377] Logged in from keytab as oryx/127.10.33.125@KRBTEST.COM (short username oryx)
I20260927 15:13:45.139796 16989 webserver.cc:460] Webserver started at http://127.10.33.125:33597/ using document root <none> and password file <none>
I20260927 15:13:45.140262 16989 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:45.140333 16989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:45.142846 16989 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:13:45.144698 17109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:45.145133 16989 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260927 15:13:45.145260 16989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/wal
uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2"
format_stamp: "Formatted at 2026-09-27 15:13:44 on dist-test-slave-f5h6"
server_key: "42418302107e8e1d33a015b3512615fc"
server_key_iv: "32680da78bb6432a73d6042710a5b7ba"
server_key_version: "encryptionkey@0"
I20260927 15:13:45.145421 16989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:45.167181 16989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:45.170394 16989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:45.189432 16989 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.125:37211
I20260927 15:13:45.189453 17160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.125:37211 every 8 connection(s)
I20260927 15:13:45.190398 16989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-1/data/info.pb
I20260927 15:13:45.196548 17161 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:13:45.200853 17161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2: Bootstrap starting.
I20260927 15:13:45.214246 17161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:13:45.214710 17161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2: Bootstrap complete.
I20260927 15:13:45.215539 17161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: NON_VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: true } }
I20260927 15:13:45.215773 17161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4bf082f24f3f46be9dcb1fddb0ff06c2, State: Initialized, Role: LEARNER
I20260927 15:13:45.215998 17161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: NON_VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: true } }
I20260927 15:13:45.216745 17164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: NON_VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: true } } }
I20260927 15:13:45.217018 17161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:13:45.217135 17164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [sys.catalog]: This master's current role is: LEARNER
I20260927 15:13:45.224308 17175 catalog_manager.cc:1259] Loaded cluster ID: 4d4d860505ed4bc6aeb9e0803aa6f40d
I20260927 15:13:45.224411 17175 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2: loading cluster ID for follower catalog manager: success
I20260927 15:13:45.227105 17175 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2: acquiring CA information for follower catalog manager: success
I20260927 15:13:45.228948 17175 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:13:45.384316 16984 heartbeater.cc:503] Master 127.10.33.126:32781 was elected leader, sending a full tablet report...
W20260927 15:13:45.438987 17086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:45.439303 17086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:45.439412 17086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:45.447207 17086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:13:45.447319 17086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:45.447387 17086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:45.447459 17086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:13:45.447516 17086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:13:45.457716 17086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:32781,127.10.33.125:37211,127.10.33.124:45567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.124.keytab
--principal=oryx/127.10.33.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.124:45567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.33.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:45.459098 17086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:45.460450 17086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:45.471935 17180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:45.471900 17181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:45.471905 17183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:45.472501 17086 server_base.cc:1034] running on GCE node
I20260927 15:13:45.473255 17086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:45.474661 17086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:45.475905 17086 hybrid_clock.cc:648] HybridClock initialized: now 1790522025475848 us; error 44 us; skew 500 ppm
I20260927 15:13:45.476259 17086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:45 dist-test-slave-f5h6 krb5kdc[16483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522025, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:45.479573 17086 init.cc:377] Logged in from keytab as oryx/127.10.33.124@KRBTEST.COM (short username oryx)
I20260927 15:13:45.481168 17086 webserver.cc:460] Webserver started at http://127.10.33.124:46007/ using document root <none> and password file <none>
I20260927 15:13:45.481891 17086 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:45.482014 17086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:45.482311 17086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:45.485343 17086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/data/instance:
uuid: "665386ca8bfb4d8f863d414ee55cba93"
format_stamp: "Formatted at 2026-09-27 15:13:45 on dist-test-slave-f5h6"
server_key: "fed628f3afbdd506dc900e526f821e88"
server_key_iv: "d0ce134b2f78012c8baae608ff1a2fce"
server_key_version: "encryptionkey@0"
I20260927 15:13:45.486157 17086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/wal/instance:
uuid: "665386ca8bfb4d8f863d414ee55cba93"
format_stamp: "Formatted at 2026-09-27 15:13:45 on dist-test-slave-f5h6"
server_key: "fed628f3afbdd506dc900e526f821e88"
server_key_iv: "d0ce134b2f78012c8baae608ff1a2fce"
server_key_version: "encryptionkey@0"
I20260927 15:13:45.491804 17086 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:13:45.495594 17190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:45.496958 17086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:13:45.497138 17086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/wal
uuid: "665386ca8bfb4d8f863d414ee55cba93"
format_stamp: "Formatted at 2026-09-27 15:13:45 on dist-test-slave-f5h6"
server_key: "fed628f3afbdd506dc900e526f821e88"
server_key_iv: "d0ce134b2f78012c8baae608ff1a2fce"
server_key_version: "encryptionkey@0"
I20260927 15:13:45.497329 17086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:45.522215 17086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:45.527042 17086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:45.527441 17086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:45.546329 17086 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.124:45567
I20260927 15:13:45.546344 17244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.124:45567 every 8 connection(s)
I20260927 15:13:45.548137 17086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/data/info.pb
I20260927 15:13:45.550275 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 17086
I20260927 15:13:45.552366 17240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.544599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56068 (local address 127.10.33.125:37211)
0927 15:13:45.545175 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:45.545181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:45.545202 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:13:45.545276 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:45.545282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:45.545369 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:45.545912 (+ 543us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:45.546050 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.547405 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.549970 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.551213 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.551392 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:45.551613 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:45.551866 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 15:13:45.552009 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":311,"threads_started":1}
I20260927 15:13:45.552645 17239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.544382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:37211 (local address 127.0.0.1:56068)
0927 15:13:45.544865 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:45.544887 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:45.545052 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:45.545566 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:45.545569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:45.545593 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:45.545768 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:45.545776 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.547595 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.547601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:45.549654 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:45.549675 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.551454 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.551462 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:45.551638 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:45.551707 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 15:13:45.551824 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 15:13:45.552223 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":217,"threads_started":1}
I20260927 15:13:45.553357 17245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:45.554154 17136 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260927 15:13:45.554714 17164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48af3088ca81428888ca24f47977bbe5. Latest consensus state: current_term: 1 leader_uuid: "48af3088ca81428888ca24f47977bbe5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: NON_VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: true } } }
I20260927 15:13:45.554978 17164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [sys.catalog]: This master's current role is: LEARNER
I20260927 15:13:45.562696 17245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:13:45 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.33.124@KRBTEST.COM
Sep 27 15:13:45 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522025, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.124@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM
I20260927 15:13:45.576993 17248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.555613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60468 (local address 127.10.33.124:45567)
0927 15:13:45.556376 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:45.556394 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:45.556449 (+ 55us) server_negotiation.cc:407] Connection header received
0927 15:13:45.557989 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:45.558012 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:45.558458 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:45.558840 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:45.560283 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.561615 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.562884 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.563600 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.563759 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:45.568546 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:45.568558 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:45.568578 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:45.568635 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:45.572734 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:45.573538 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:45.573545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:45.573553 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:45.573783 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:45.574237 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:45.574242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:45.574245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:45.574818 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:45.575057 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:45.575507 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 15:13:45.575855 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":320,"threads_started":1}
I20260927 15:13:45.580302 17249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.565066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.0.0.1:54414)
0927 15:13:45.565701 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:45.565761 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:45.565943 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:45.566289 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:45.566296 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:45.566865 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:45.567092 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:45.567100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.568417 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.568422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:45.570106 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:45.570125 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.570973 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.570980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:45.571186 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:45.571744 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:45.571766 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:45.574266 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:45.577615 (+ 3349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:45.577622 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:45.577627 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:45.578085 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:45.578493 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:45.578497 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:45.578501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:45.578665 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:45.579237 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:45.579244 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:45.579427 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 15:13:45.579616 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 15:13:45.579922 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":305,"threads_started":1}
I20260927 15:13:45.580385 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.565179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54414 (local address 127.10.33.126:32781)
0927 15:13:45.565356 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:45.565361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:45.565851 (+ 490us) server_negotiation.cc:407] Connection header received
0927 15:13:45.566058 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:45.566063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:45.566126 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:45.566204 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:45.567242 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.568256 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.570343 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.570774 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.570871 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:45.574451 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:45.574456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:45.574459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:45.574491 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:45.577486 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:45.578215 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:45.578221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:45.578224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:45.578386 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:45.578810 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:45.578814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:45.578816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:45.579109 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:45.579231 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:45.579758 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 15:13:45.580018 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:13:45.581634 17245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:13:45 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522025, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.124@KRBTEST.COM for oryx/127.10.33.125@KRBTEST.COM
I20260927 15:13:45.595064 17240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.582498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56070 (local address 127.10.33.125:37211)
0927 15:13:45.582698 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:45.582705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:45.582723 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:13:45.582773 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:45.582777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:45.582838 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:45.582929 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:45.583636 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.584390 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.585697 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.586348 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.586484 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:45.589635 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:45.589640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:45.589643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:45.589690 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:45.592471 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:45.593179 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:45.593183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:45.593186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:45.593355 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:45.593703 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:45.593707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:45.593709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:45.594041 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:45.594173 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:45.594510 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 15:13:45.594730 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:13:45.595059 17249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.582297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:37211 (local address 127.0.0.1:56070)
0927 15:13:45.582476 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:45.582493 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:45.582593 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:45.582964 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:45.582969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:45.583332 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:45.583472 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:45.583481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.584551 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.584555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:45.585539 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:45.585553 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.586518 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.586525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:45.586701 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:45.587324 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:45.587342 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:45.589432 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:45.592642 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:45.592646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:45.592649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:45.593054 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:45.593467 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:45.593471 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:45.593473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:45.593601 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:45.594156 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:45.594161 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:45.594293 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 15:13:45.594409 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 15:13:45.594711 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 15:13:45.596081 17245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.124" port: 45567 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:13:45 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522025, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.124@KRBTEST.COM for oryx/127.10.33.124@KRBTEST.COM
I20260927 15:13:45.608479 17249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.596916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:45567 (local address 127.0.0.1:60484)
0927 15:13:45.597167 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:45.597180 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:45.597311 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:45.597552 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:45.597555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:45.597907 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:45.598058 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:45.598067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.599033 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.599037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:45.599929 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:45.599941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.600656 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.600661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:45.600828 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:45.601390 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:45.601405 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:45.603662 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:45.606279 (+ 2617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:45.606284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:45.606289 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:45.606608 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:45.606927 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:45.606932 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:45.606935 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:45.607066 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:45.607593 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:45.607598 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:45.607726 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 15:13:45.607844 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 15:13:45.608087 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260927 15:13:45.608543 17248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.596911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60484 (local address 127.10.33.124:45567)
0927 15:13:45.597084 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:45.597088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:45.597293 (+ 205us) server_negotiation.cc:407] Connection header received
0927 15:13:45.597394 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:45.597397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:45.597452 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:45.597530 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:45.598224 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.598915 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.600066 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.600475 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.600598 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:45.603849 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:45.603854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:45.603857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:45.603884 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:45.606160 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:45.606710 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:45.606715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:45.606719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:45.606826 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:45.607168 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:45.607173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:45.607176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:45.607492 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:45.607621 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:45.607930 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 15:13:45.608231 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":66}
I20260927 15:13:45.613293 17245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93: Bootstrap starting.
I20260927 15:13:45.616467 17245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:45.617594 17245 log.cc:824] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:45.620951 17245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93: No bootstrap required, opened a new log
I20260927 15:13:45.627974 17245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 45567 } }
I20260927 15:13:45.628321 17245 raft_consensus.cc:374] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:45.628424 17245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 665386ca8bfb4d8f863d414ee55cba93, State: Initialized, Role: FOLLOWER
I20260927 15:13:45.628975 17245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 45567 } }
I20260927 15:13:45.629823 17251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 45567 } } }
I20260927 15:13:45.630156 17251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:13:45.630942 17245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 15:13:45.640211 17262 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:13:45.640372 17262 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:13:45.644389 17086 master_runner.cc:428] Detected that this master 665386ca8bfb4d8f863d414ee55cba93 is joining an existing cluster
I20260927 15:13:45.644596 17086 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.33.124:45567
I20260927 15:13:45.666375 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.649120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54418 (local address 127.10.33.126:32781)
0927 15:13:45.649336 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:45.649342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:45.650402 (+ 1060us) server_negotiation.cc:407] Connection header received
0927 15:13:45.650586 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:45.650593 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:45.650694 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:45.650912 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:45.652169 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.653407 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.655740 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.656476 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.656578 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:45.659548 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:45.659555 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:45.659561 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:45.659600 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:45.662987 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:45.663814 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:45.663820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:45.663824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:45.663977 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:45.664465 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:45.664470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:45.664472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:45.664845 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:45.664977 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:45.665713 (+ 736us) server_negotiation.cc:299] Negotiation successful
0927 15:13:45.666013 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 15:13:45.667889 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.649164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.0.0.1:54418)
0927 15:13:45.650269 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:45.650289 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:45.650452 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:45.650927 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:45.650933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:45.651610 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:45.651989 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:45.652002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.653576 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.653582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:45.655554 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:45.655570 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.656876 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.656882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:45.657062 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:45.657842 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:45.657885 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:45.659291 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:45.663194 (+ 3903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:45.663198 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:45.663202 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:45.663666 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:45.664127 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:45.664132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:45.664136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:45.664334 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:45.665054 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:45.665065 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:45.665401 (+ 336us) client_negotiation.cc:769] Sending connection context
0927 15:13:45.666950 (+ 1549us) client_negotiation.cc:240] Negotiation successful
0927 15:13:45.667297 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"thread_start_us":4927,"threads_started":1}
I20260927 15:13:45.672505 17248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.649142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60488 (local address 127.10.33.124:45567)
0927 15:13:45.649343 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:45.649348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:45.654523 (+ 5175us) server_negotiation.cc:407] Connection header received
0927 15:13:45.654757 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:45.654762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:45.654826 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:45.654886 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:45.656373 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.657527 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.660081 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.660800 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.660891 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:45.665008 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:45.665015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:45.665019 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:45.665060 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:45.668891 (+ 3831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:45.669676 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:45.669682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:45.669686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:45.669897 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:45.670285 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:45.670289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:45.670292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:45.670735 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:45.670885 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:45.671911 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0927 15:13:45.672143 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:13:45.673614 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.649575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:45567 (local address 127.0.0.1:60488)
0927 15:13:45.654434 (+ 4859us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:45.654455 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:45.654630 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:45.655079 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:45.655084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:45.655857 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:45.656222 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:45.656235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.658178 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.658182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:45.659931 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:45.659947 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.661179 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.661185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:45.661350 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:45.662037 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:45.662065 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:45.664750 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:45.669046 (+ 4296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:45.669054 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:45.669058 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:45.669563 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:45.670021 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:45.670025 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:45.670029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:45.670179 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:45.671388 (+ 1209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:45.671394 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:45.671612 (+ 218us) client_negotiation.cc:769] Sending connection context
0927 15:13:45.672755 (+ 1143us) client_negotiation.cc:240] Negotiation successful
0927 15:13:45.673086 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4723,"thread_start_us":378,"threads_started":1}
I20260927 15:13:45.681648 17240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.654663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56078 (local address 127.10.33.125:37211)
0927 15:13:45.654840 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:45.654845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:45.665709 (+ 10864us) server_negotiation.cc:407] Connection header received
0927 15:13:45.667224 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:45.667231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:45.667331 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:45.667410 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:45.669144 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.670223 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.672304 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.672933 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.673039 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:45.676043 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:45.676052 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:45.676057 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:45.676105 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:45.679009 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:45.679677 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:45.679683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:45.679686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:45.679823 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:45.680201 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:45.680213 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:45.680215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:45.680473 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:45.680605 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:45.681089 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 15:13:45.681327 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 15:13:45.682602 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.654969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:37211 (local address 127.0.0.1:56078)
0927 15:13:45.665588 (+ 10619us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:45.665612 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:45.667042 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:45.667620 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:45.667625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:45.668612 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:45.668966 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:45.668978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.670393 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.670398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:45.672133 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:45.672146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.673618 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.673625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:45.673821 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:45.674780 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:45.674801 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:45.675807 (+ 1006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:45.679152 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:45.679157 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:45.679162 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:45.679558 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:45.679929 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:45.679934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:45.679938 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:45.680097 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:45.680622 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:45.680627 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:45.680812 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 15:13:45.681896 (+ 1084us) client_negotiation.cc:240] Negotiation successful
0927 15:13:45.682192 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10442,"mutex_wait_us":2,"thread_start_us":10766,"threads_started":1}
I20260927 15:13:45.683825 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 16589
Sep 27 15:13:45 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522022, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.126@KRBTEST.COM for oryx/127.10.33.124@KRBTEST.COM
I20260927 15:13:45.692345 17239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.674342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:45567 (local address 127.0.0.1:60498)
0927 15:13:45.674499 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:45.674516 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:45.674616 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:45.676081 (+ 1465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:45.676086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:45.676496 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:45.676645 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:45.676653 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.678297 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.678304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:45.679779 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:45.679798 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.681009 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.681015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:45.681189 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:45.682203 (+ 1014us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:45.682234 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:45.685573 (+ 3339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:45.689325 (+ 3752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:45.689331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:45.689336 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:45.689746 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:45.690227 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:45.690232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:45.690235 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:45.690437 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:45.691124 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:45.691129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:45.691302 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 15:13:45.691454 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 15:13:45.691735 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 15:13:45.693080 17248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.675552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60498 (local address 127.10.33.124:45567)
0927 15:13:45.675732 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:45.675738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:45.675755 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:13:45.675847 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:45.675852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:45.675936 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:45.676014 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:45.676812 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.678083 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.680100 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.680830 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.680991 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:45.685805 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:45.685811 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:45.685815 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:45.685851 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:45.689159 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:45.689922 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:45.689927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:45.689931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:45.690075 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:45.690599 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:45.690604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:45.690608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:45.690977 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:45.692575 (+ 1598us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:45.692683 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 15:13:45.692859 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:13:45.694499 16528 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.33.124:45567: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93"
member_type: NON_VOTER
last_known_addr {
host: "127.10.33.124"
port: 45567
}
attrs {
promote: true
}
}
dest_uuid: "48af3088ca81428888ca24f47977bbe5"
cas_config_opid_index: 5
I20260927 15:13:45.695550 16528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: NON_VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: true } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: NON_VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: true } }
I20260927 15:13:45.697319 17233 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 665386ca8bfb4d8f863d414ee55cba93 (127.10.33.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: NON_VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: true } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: NON_VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: true } } }
I20260927 15:13:45.697992 17136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 1 LEARNER]: Refusing update from remote peer 48af3088ca81428888ca24f47977bbe5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 15:13:45.698964 17217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 15:13:45 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.33.65@KRBTEST.COM
I20260927 15:13:45.702687 17217 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [term 1 FOLLOWER]: Refusing update from remote peer 48af3088ca81428888ca24f47977bbe5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 15:13:45.709940 17233 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.33.124:45567
I20260927 15:13:45.710626 17086 master.cc:537] Master@127.10.33.124:45567 shutting down...
I20260927 15:13:45.712350 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.685303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55172 (local address 127.10.33.65:41861)
0927 15:13:45.686150 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:45.686160 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:45.686243 (+ 83us) server_negotiation.cc:407] Connection header received
0927 15:13:45.686323 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:45.686332 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:45.686758 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:45.686945 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:45.688602 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.690321 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.692154 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.693498 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.693654 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:45.703386 (+ 9732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:45.703409 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:45.703430 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:45.703483 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:45.707729 (+ 4246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:45.708822 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:45.708833 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:45.708837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:45.709009 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:45.709494 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:45.709498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:45.709501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:45.709981 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:45.711487 (+ 1506us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:45.711658 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 15:13:45.711951 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":367,"threads_started":1}
I20260927 15:13:45.714068 16712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:45.709846 17273 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: NON_VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20260927 15:13:45.728570 17136 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 665386ca8bfb4d8f863d414ee55cba93 (127.10.33.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: NON_VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: true } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: NON_VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: true } } }
I20260927 15:13:45.731607 17272 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: NON_VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:45.732206 17274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "48af3088ca81428888ca24f47977bbe5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: NON_VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: true } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: NON_VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: true } } }
I20260927 15:13:45.732539 17274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:45.733139 17164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "48af3088ca81428888ca24f47977bbe5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: NON_VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: true } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: NON_VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: true } } }
I20260927 15:13:45.733421 17164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [sys.catalog]: This master's current role is: LEARNER
I20260927 15:13:45.735901 17233 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5: attempting to promote NON_VOTER 4bf082f24f3f46be9dcb1fddb0ff06c2 to VOTER
I20260927 15:13:45.738119 17233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: NON_VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: true } }
I20260927 15:13:45.740909 17136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 1 LEARNER]: Refusing update from remote peer 48af3088ca81428888ca24f47977bbe5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 15:13:45.741483 17233 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 15:13:45.743758 17233 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4bf082f24f3f46be9dcb1fddb0ff06c2 (127.10.33.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: NON_VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: true } } }
I20260927 15:13:45.748227 17233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "48af3088ca81428888ca24f47977bbe5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: NON_VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: true } } }
I20260927 15:13:45.748500 17233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:45.748756 17233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "48af3088ca81428888ca24f47977bbe5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: NON_VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: true } } }
I20260927 15:13:45.748973 17233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:45.749154 17135 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4bf082f24f3f46be9dcb1fddb0ff06c2 (127.10.33.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: NON_VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: true } } }
I20260927 15:13:45.754244 17086 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:45.754710 17086 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:45.754947 17086 tablet_replica.cc:330] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93: stopping tablet replica
W20260927 15:13:45.755854 16515 consensus_peers.cc:487] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 -> Peer 665386ca8bfb4d8f863d414ee55cba93 (127.10.33.124:45567): Couldn't send request to peer 665386ca8bfb4d8f863d414ee55cba93. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 15:13:45.756503 17164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "48af3088ca81428888ca24f47977bbe5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: NON_VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: true } } }
I20260927 15:13:45.756795 17164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:13:45.761438 17086 master.cc:552] Master@127.10.33.124:45567 shutdown complete.
I20260927 15:13:45.761533 17086 master_runner.cc:305] Clearing existing system tablet
I20260927 15:13:45.761926 17086 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:45.762030 17086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:45.764842 17086 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 15:13:45.766459 17280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:45.766804 17086 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:45.767232 17086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:13:45.767373 17086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/wal
uuid: "665386ca8bfb4d8f863d414ee55cba93"
format_stamp: "Formatted at 2026-09-27 15:13:45 on dist-test-slave-f5h6"
server_key: "fed628f3afbdd506dc900e526f821e88"
server_key_iv: "d0ce134b2f78012c8baae608ff1a2fce"
server_key_version: "encryptionkey@0"
I20260927 15:13:45.772517 17086 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:45.779645 17086 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 15:13:45.779906 17086 log.cc:1196] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 15:13:45.780421 17086 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93: Deleting consensus metadata
I20260927 15:13:45.780686 17086 master_runner.cc:315] Copying system tablet from 127.10.33.126:32781
I20260927 15:13:45.783264 17086 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93: tablet copy: Beginning tablet copy session from remote peer at address 127.10.33.126:32781
I20260927 15:13:45.799136 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.784176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54422 (local address 127.10.33.126:32781)
0927 15:13:45.784413 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:45.784419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:45.784892 (+ 473us) server_negotiation.cc:407] Connection header received
0927 15:13:45.785150 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:45.785157 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:45.785231 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:45.785297 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:45.786469 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.787621 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.789562 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.790243 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.790358 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:45.792802 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:45.792809 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:45.792814 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:45.792854 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:45.796245 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:45.797033 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:45.797038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:45.797042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:45.797202 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:45.797591 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:45.797595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:45.797598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:45.797949 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:45.798099 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:45.798532 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 15:13:45.798772 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 15:13:45.799206 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:45.784139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.0.0.1:54422)
0927 15:13:45.784772 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:45.784813 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:45.785009 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:45.785490 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:45.785495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:45.786046 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:45.786306 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:45.786317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.787813 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.787818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:45.789404 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:45.789419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:45.790484 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:45.790491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:45.790674 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:45.791417 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:45.791446 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:45.792564 (+ 1118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:45.796404 (+ 3840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:45.796410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:45.796414 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:45.796911 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:45.797314 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:45.797319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:45.797323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:45.797494 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:45.798073 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:45.798077 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:45.798268 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 15:13:45.798433 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 15:13:45.798753 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":368,"threads_started":1}
I20260927 15:13:45.800158 16548 tablet_copy_service.cc:140] P 48af3088ca81428888ca24f47977bbe5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 665386ca8bfb4d8f863d414ee55cba93 ({username='oryx', principal='oryx/127.10.33.124@KRBTEST.COM'} at 127.0.0.1:54422)
I20260927 15:13:45.800320 16548 tablet_copy_service.cc:161] P 48af3088ca81428888ca24f47977bbe5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 665386ca8bfb4d8f863d414ee55cba93 at {username='oryx', principal='oryx/127.10.33.124@KRBTEST.COM'} at 127.0.0.1:54422: session id = 665386ca8bfb4d8f863d414ee55cba93-00000000000000000000000000000000
I20260927 15:13:45.804075 16548 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 15:13:45.805985 17086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 15:13:45.813900 17086 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93: tablet copy: Starting download of 0 data blocks...
I20260927 15:13:45.814211 17086 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93: tablet copy: Starting download of 1 WAL segments...
I20260927 15:13:45.816723 17086 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 15:13:45.821161 16548 tablet_copy_service.cc:342] P 48af3088ca81428888ca24f47977bbe5: Request end of tablet copy session 665386ca8bfb4d8f863d414ee55cba93-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.33.124@KRBTEST.COM'} at 127.0.0.1:54422
I20260927 15:13:45.821314 16548 tablet_copy_service.cc:433] P 48af3088ca81428888ca24f47977bbe5: ending tablet copy session 665386ca8bfb4d8f863d414ee55cba93-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 665386ca8bfb4d8f863d414ee55cba93
I20260927 15:13:45.833029 17086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:45.835788 17287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:45.843982 17288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:45.847162 17086 server_base.cc:1034] running on GCE node
W20260927 15:13:45.847925 17290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:45.848456 17086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:45.849180 17086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:45.850406 17086 hybrid_clock.cc:648] HybridClock initialized: now 1790522025850378 us; error 50 us; skew 500 ppm
I20260927 15:13:45.850641 17086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:45 dist-test-slave-f5h6 krb5kdc[16483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522025, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:45.852706 17086 init.cc:377] Logged in from keytab as oryx/127.10.33.124@KRBTEST.COM (short username oryx)
I20260927 15:13:45.862205 17086 webserver.cc:460] Webserver started at http://127.10.33.124:44151/ using document root <none> and password file <none>
I20260927 15:13:45.862577 17086 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:45.862686 17086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:45.866467 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 16589
I20260927 15:13:45.866519 17086 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:13:45.868415 17297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:45.868777 17086 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 15:13:45.868881 17086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/wal
uuid: "665386ca8bfb4d8f863d414ee55cba93"
format_stamp: "Formatted at 2026-09-27 15:13:45 on dist-test-slave-f5h6"
server_key: "fed628f3afbdd506dc900e526f821e88"
server_key_iv: "d0ce134b2f78012c8baae608ff1a2fce"
server_key_version: "encryptionkey@0"
I20260927 15:13:45.869015 17086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:45.890733 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:41861
--local_ip_for_outbound_sockets=127.10.33.65
--tserver_master_addrs=127.10.33.126:32781
--webserver_port=46519
--webserver_interface=127.10.33.65
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.65.keytab
--principal=oryx/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.33.126:32781,127.10.33.125:37211
--tserver_master_addrs=127.10.33.126:32781,127.10.33.125:37211,127.10.33.124:45567 with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:13:45.915814 17086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:45.920290 17086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:45.966629 17086 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.124:45567
I20260927 15:13:45.966693 17349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.124:45567 every 8 connection(s)
I20260927 15:13:45.967765 17086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/master-2/data/info.pb
I20260927 15:13:45.975155 17350 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:13:45.979465 17350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93: Bootstrap starting.
I20260927 15:13:46.015041 17350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:13:46.015595 17350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93: Bootstrap complete.
I20260927 15:13:46.017391 17350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: NON_VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: true } }
I20260927 15:13:46.017813 17350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 665386ca8bfb4d8f863d414ee55cba93, State: Initialized, Role: LEARNER
I20260927 15:13:46.018119 17350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: NON_VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: true } }
I20260927 15:13:46.019104 17353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: NON_VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: true } } }
I20260927 15:13:46.019713 17353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [sys.catalog]: This master's current role is: LEARNER
I20260927 15:13:46.019536 17350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:13:46.029505 17364 catalog_manager.cc:1259] Loaded cluster ID: 4d4d860505ed4bc6aeb9e0803aa6f40d
I20260927 15:13:46.029630 17364 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93: loading cluster ID for follower catalog manager: success
I20260927 15:13:46.033388 17364 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93: acquiring CA information for follower catalog manager: success
I20260927 15:13:46.036118 17364 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:13:46.260912 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:46.253775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60502 (local address 127.10.33.124:45567)
0927 15:13:46.254396 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:46.254402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:46.254426 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:13:46.254501 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:46.254507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:46.254620 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:46.254752 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:46.255371 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:46.256881 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:46.259079 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:46.260027 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:46.260138 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:46.260293 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:46.260534 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 15:13:46.260638 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":374,"threads_started":1}
I20260927 15:13:46.260968 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:46.253697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:45567 (local address 127.0.0.1:60502)
0927 15:13:46.254208 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:46.254227 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:46.254357 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:46.254805 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:46.254809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:46.254833 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:46.255208 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:46.255227 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:46.257077 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:46.257082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:46.258903 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:46.258920 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:46.260206 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:46.260212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:46.260379 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:46.260414 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 15:13:46.260515 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 15:13:46.260664 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":222,"threads_started":1}
I20260927 15:13:46.261749 17325 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 15:13:46.262279 17353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48af3088ca81428888ca24f47977bbe5. Latest consensus state: current_term: 1 leader_uuid: "48af3088ca81428888ca24f47977bbe5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: NON_VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: true } } }
I20260927 15:13:46.262559 17353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [sys.catalog]: This master's current role is: LEARNER
W20260927 15:13:46.281736 17302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:46.282092 17302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:46.282236 17302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:46.289387 17302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:46.289532 17302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:46.289655 17302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:13:46.299319 17302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.65.keytab
--principal=oryx/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:41861
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=46519
--tserver_master_addrs=127.10.33.126:32781,127.10.33.125:37211,127.10.33.124:45567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:46.300716 17302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:46.301997 17302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:46.315047 17371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:46.315222 17372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:46.315351 17374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:46.315927 17302 server_base.cc:1034] running on GCE node
I20260927 15:13:46.316607 17302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:46.317612 17302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:46.318845 17302 hybrid_clock.cc:648] HybridClock initialized: now 1790522026318775 us; error 61 us; skew 500 ppm
I20260927 15:13:46.319211 17302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:46 dist-test-slave-f5h6 krb5kdc[16483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522026, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:46.322399 17302 init.cc:377] Logged in from keytab as oryx/127.10.33.65@KRBTEST.COM (short username oryx)
I20260927 15:13:46.324061 17302 webserver.cc:460] Webserver started at http://127.10.33.65:46519/ using document root <none> and password file <none>
I20260927 15:13:46.324762 17302 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:46.324904 17302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:46.330909 17302 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:13:46.333601 17381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:46.335040 17302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:13:46.335198 17302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3"
format_stamp: "Formatted at 2026-09-27 15:13:43 on dist-test-slave-f5h6"
server_key: "8bcdaf4efcfbdd3e090fbf0f7ccc5f57"
server_key_iv: "a6c4ad9e88d1c000e414b1c8e1ea9d02"
server_key_version: "encryptionkey@0"
I20260927 15:13:46.336159 17302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:46.359558 17302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:46.363955 17302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:46.364236 17302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:46.365370 17302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:46.366860 17302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:46.366955 17302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:46.367057 17302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:46.367250 17302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:46.397362 17302 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:41861
I20260927 15:13:46.397435 17493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:41861 every 8 connection(s)
I20260927 15:13:46.398782 17302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:13:46.406140 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 17302
I20260927 15:13:46.406510 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 16723
Sep 27 15:13:46 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522026, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.124@KRBTEST.COM
Sep 27 15:13:46 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522026, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM
Sep 27 15:13:46 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522026, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.125@KRBTEST.COM
I20260927 15:13:46.428404 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:46.404116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:37877 (local address 127.10.33.125:37211)
0927 15:13:46.404657 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:46.404661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:46.404686 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:13:46.405797 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:46.405803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:46.405915 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:46.406017 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:46.408797 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:46.410102 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:46.414605 (+ 4503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:46.415317 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:46.415436 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:46.421592 (+ 6156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:46.421599 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:46.421604 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:46.421664 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:46.424920 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:46.426076 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:46.426081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:46.426085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:46.426233 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:46.426891 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:46.426897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:46.426901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:46.427249 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:46.427367 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:46.427816 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 15:13:46.428027 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":306,"threads_started":1}
I20260927 15:13:46.428427 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:46.402176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:38905 (local address 127.10.33.124:45567)
0927 15:13:46.402391 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:46.402399 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:46.403480 (+ 1081us) server_negotiation.cc:407] Connection header received
0927 15:13:46.405739 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:46.405746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:46.405835 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:46.405940 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:46.408817 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:46.410135 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:46.413270 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:46.414063 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:46.414182 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:46.421205 (+ 7023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:46.421212 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:46.421218 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:46.421266 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:46.424868 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:46.426005 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:46.426010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:46.426015 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:46.426148 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:46.426571 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:46.426575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:46.426577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:46.426845 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:46.426983 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:46.427866 (+ 883us) server_negotiation.cc:299] Negotiation successful
0927 15:13:46.428098 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:13:46.429391 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:46.403913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:37211 (local address 127.10.33.65:37877)
0927 15:13:46.404440 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:46.404461 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:46.405422 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:46.406387 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:46.406399 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:46.407384 (+ 985us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:46.408554 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:46.408578 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:46.410423 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:46.410427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:46.414435 (+ 4008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:46.414449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:46.416283 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:46.416289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:46.416490 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:46.418348 (+ 1858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:46.418371 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:46.421338 (+ 2967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:46.425083 (+ 3745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:46.425087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:46.425091 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:46.425922 (+ 831us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:46.426605 (+ 683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:46.426609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:46.426612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:46.426766 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:46.427369 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:46.427373 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:46.427506 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 15:13:46.427702 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 15:13:46.428203 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":2510,"spinlock_wait_cycles":768,"thread_start_us":276,"threads_started":1}
I20260927 15:13:46.429423 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:46.402340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:45567 (local address 127.10.33.65:38905)
0927 15:13:46.403314 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:46.403358 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:46.405435 (+ 2077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:46.406392 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:46.406400 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:46.407389 (+ 989us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:46.408550 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:46.408579 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:46.410382 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:46.410389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:46.413023 (+ 2634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:46.413062 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:46.414623 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:46.414631 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:46.414995 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:46.415930 (+ 935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:46.415968 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:46.420907 (+ 4939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:46.425023 (+ 4116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:46.425034 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:46.425055 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:46.425892 (+ 837us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:46.426264 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:46.426267 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:46.426269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:46.426451 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:46.426953 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:46.426962 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:46.427403 (+ 441us) client_negotiation.cc:769] Sending connection context
0927 15:13:46.428688 (+ 1285us) client_negotiation.cc:240] Negotiation successful
0927 15:13:46.428968 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":64,"spinlock_wait_cycles":60800,"thread_start_us":367,"threads_started":1}
I20260927 15:13:46.431025 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:46.402340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.10.33.65:38403)
0927 15:13:46.403312 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:46.403357 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:46.405405 (+ 2048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:46.406387 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:46.406399 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:46.407610 (+ 1211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:46.410291 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:46.410305 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:46.411810 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:46.411815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:46.413476 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:46.413492 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:46.415697 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:46.415704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:46.415923 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:46.416880 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:46.416902 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:46.420502 (+ 3600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:46.425297 (+ 4795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:46.425302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:46.425306 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:46.426487 (+ 1181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:46.427066 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:46.427071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:46.427075 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:46.427229 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:46.430015 (+ 2786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:46.430020 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:46.430183 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 15:13:46.430323 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 15:13:46.430607 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":2283,"spinlock_wait_cycles":1280,"thread_start_us":354,"threads_started":1}
I20260927 15:13:46.431165 17494 heartbeater.cc:347] Connected to a master server at 127.10.33.124:45567
I20260927 15:13:46.431320 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:46.401693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:38403 (local address 127.10.33.126:32781)
0927 15:13:46.402700 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:46.402707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:46.403471 (+ 764us) server_negotiation.cc:407] Connection header received
0927 15:13:46.405711 (+ 2240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:46.405717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:46.405800 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:46.405914 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:46.410424 (+ 4510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:46.411507 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:46.413609 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:46.414300 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:46.414419 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:46.420811 (+ 6392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:46.420819 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:46.420824 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:46.420866 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:46.425129 (+ 4263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:46.426598 (+ 1469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:46.426603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:46.426606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:46.426753 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:46.427339 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:46.427344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:46.427348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:46.427673 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:46.427789 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:46.430469 (+ 2680us) server_negotiation.cc:299] Negotiation successful
0927 15:13:46.430707 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":879,"thread_start_us":779,"threads_started":1}
I20260927 15:13:46.431486 17494 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:46.431129 17496 heartbeater.cc:347] Connected to a master server at 127.10.33.125:37211
I20260927 15:13:46.431957 17496 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:46.432042 17495 heartbeater.cc:347] Connected to a master server at 127.10.33.126:32781
I20260927 15:13:46.432194 17495 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:46.432484 17496 heartbeater.cc:511] Master 127.10.33.125:37211 requested a full tablet report, sending...
I20260927 15:13:46.432518 17494 heartbeater.cc:511] Master 127.10.33.124:45567 requested a full tablet report, sending...
I20260927 15:13:46.432690 17495 heartbeater.cc:511] Master 127.10.33.126:32781 requested a full tablet report, sending...
I20260927 15:13:46.434351 16528 ts_manager.cc:194] Re-registered known tserver with Master: 48e1d19f3efd444fa8f0ba4fea3b7da3 (127.10.33.65:41861)
I20260927 15:13:46.434840 17126 ts_manager.cc:194] Registered new tserver with Master: 48e1d19f3efd444fa8f0ba4fea3b7da3 (127.10.33.65:41861)
I20260927 15:13:46.434875 17315 ts_manager.cc:194] Registered new tserver with Master: 48e1d19f3efd444fa8f0ba4fea3b7da3 (127.10.33.65:41861)
I20260927 15:13:46.435359 16528 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:38403
Sep 27 15:13:46 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.33.66@KRBTEST.COM
I20260927 15:13:46.447208 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:46.407421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53870 (local address 127.10.33.66:37121)
0927 15:13:46.426169 (+ 18748us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:46.426178 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:46.426207 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:13:46.426280 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:46.426290 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:46.426905 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:46.427071 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:46.431877 (+ 4806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:46.434629 (+ 2752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:46.436342 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:46.436880 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:46.436992 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:46.441402 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:46.441414 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:46.441427 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:46.441461 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:46.444350 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:46.445105 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:46.445112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:46.445115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:46.445257 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:46.445634 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:46.445637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:46.445640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:46.446054 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:46.446296 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:46.446701 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 15:13:46.446914 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18571,"thread_start_us":18180,"threads_started":1}
I20260927 15:13:46.448652 16846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:46.530887 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 16723
I20260927 15:13:46.544864 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:37121
--local_ip_for_outbound_sockets=127.10.33.66
--tserver_master_addrs=127.10.33.126:32781
--webserver_port=45045
--webserver_interface=127.10.33.66
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.66.keytab
--principal=oryx/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.33.126:32781,127.10.33.125:37211
--tserver_master_addrs=127.10.33.126:32781,127.10.33.125:37211,127.10.33.124:45567 with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:13:46.663170 17273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "48af3088ca81428888ca24f47977bbe5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: NON_VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: true } } }
I20260927 15:13:46.663570 17273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:46.664951 17507 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5: attempting to promote NON_VOTER 665386ca8bfb4d8f863d414ee55cba93 to VOTER
I20260927 15:13:46.665498 17507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: false } }
I20260927 15:13:46.667546 17325 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [term 1 LEARNER]: Refusing update from remote peer 48af3088ca81428888ca24f47977bbe5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 15:13:46.667557 17135 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 1 FOLLOWER]: Refusing update from remote peer 48af3088ca81428888ca24f47977bbe5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 15:13:46.668134 17508 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 15:13:46.668558 17273 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 15:13:46.671550 17273 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 665386ca8bfb4d8f863d414ee55cba93 (127.10.33.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: false } } }
I20260927 15:13:46.672192 17135 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 665386ca8bfb4d8f863d414ee55cba93 (127.10.33.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: false } } }
I20260927 15:13:46.675567 17273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "48af3088ca81428888ca24f47977bbe5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: false } } }
I20260927 15:13:46.675834 17273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:46.676083 17273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "48af3088ca81428888ca24f47977bbe5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: false } } }
I20260927 15:13:46.676324 17273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:46.677074 17511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "48af3088ca81428888ca24f47977bbe5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: false } } }
I20260927 15:13:46.677472 17511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:13:46.678172 17507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "48af3088ca81428888ca24f47977bbe5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: false } } }
I20260927 15:13:46.678500 17507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:46.678501 17325 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 665386ca8bfb4d8f863d414ee55cba93 (127.10.33.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: false } } }
I20260927 15:13:46.683072 17353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "48af3088ca81428888ca24f47977bbe5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: false } } }
I20260927 15:13:46.683326 17353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 15:13:46.886368 17506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:46.886668 17506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:46.886759 17506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:46.894824 17506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:46.894963 17506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:46.895077 17506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:13:46.905692 17506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.66.keytab
--principal=oryx/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:37121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=45045
--tserver_master_addrs=127.10.33.126:32781,127.10.33.125:37211,127.10.33.124:45567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:46.907030 17506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:46.908288 17506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:46.922027 17518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:46.922075 17517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:46.922075 17520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:46.923156 17506 server_base.cc:1034] running on GCE node
I20260927 15:13:46.923715 17506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:46.924834 17506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:46.926124 17506 hybrid_clock.cc:648] HybridClock initialized: now 1790522026926019 us; error 63 us; skew 500 ppm
I20260927 15:13:46.926453 17506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:46 dist-test-slave-f5h6 krb5kdc[16483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522026, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:46.929508 17506 init.cc:377] Logged in from keytab as oryx/127.10.33.66@KRBTEST.COM (short username oryx)
I20260927 15:13:46.931003 17506 webserver.cc:460] Webserver started at http://127.10.33.66:45045/ using document root <none> and password file <none>
I20260927 15:13:46.931749 17506 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:46.931890 17506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:46.937698 17506 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 15:13:46.941099 17527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:46.942674 17506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:13:46.942849 17506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "49233ab4d77a4c58b34c9f40d42ff227"
format_stamp: "Formatted at 2026-09-27 15:13:43 on dist-test-slave-f5h6"
server_key: "89e72945678b9ce155eaa375576b6e21"
server_key_iv: "91112c6990af562f48a4cb5e9c462db1"
server_key_version: "encryptionkey@0"
I20260927 15:13:46.943847 17506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:46.971966 17506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:46.977048 17506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:46.977346 17506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:46.978545 17506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:46.979996 17506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:46.980084 17506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:46.980182 17506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:46.980253 17506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:47.011415 17506 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:37121
I20260927 15:13:47.011487 17639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:37121 every 8 connection(s)
I20260927 15:13:47.012826 17506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:13:47.015969 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 17506
I20260927 15:13:47.016281 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 16856
Sep 27 15:13:47 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.33.67@KRBTEST.COM
Sep 27 15:13:47 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522026, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM
Sep 27 15:13:47 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522026, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.124@KRBTEST.COM
Sep 27 15:13:47 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522026, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.125@KRBTEST.COM
I20260927 15:13:47.053333 17649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.017712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51694 (local address 127.10.33.67:34725)
0927 15:13:47.026677 (+ 8965us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:47.026689 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:47.032147 (+ 5458us) server_negotiation.cc:407] Connection header received
0927 15:13:47.032354 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:47.032365 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:47.032639 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:47.032803 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:47.034845 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.036306 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.038094 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.038896 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.039011 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.045680 (+ 6669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:47.045696 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:47.045714 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:47.045758 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:47.050347 (+ 4589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:47.051196 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:47.051206 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:47.051209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:47.051378 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:47.051755 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:47.051759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:47.051762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:47.052114 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:47.052439 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:47.052667 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 15:13:47.052908 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8784,"thread_start_us":8639,"threads_started":1}
I20260927 15:13:47.054617 16979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:47.056905 17646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.016318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:45567 (local address 127.10.33.66:45827)
0927 15:13:47.026596 (+ 10278us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:47.026668 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:47.029344 (+ 2676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:47.038625 (+ 9281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:47.038631 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:47.039217 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:47.039512 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:47.039535 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.041356 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.041362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:47.043014 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:47.043056 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.044210 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.044226 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:47.044542 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.045188 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:47.045212 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:47.047632 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:47.051343 (+ 3711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:47.051347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:47.051351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:47.052944 (+ 1593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:47.053418 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:47.053422 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:47.053426 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:47.053656 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:47.054355 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:47.054364 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:47.054742 (+ 378us) client_negotiation.cc:769] Sending connection context
0927 15:13:47.055025 (+ 283us) client_negotiation.cc:240] Negotiation successful
0927 15:13:47.055535 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9854,"thread_start_us":13439,"threads_started":1}
I20260927 15:13:47.058722 17645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.016256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:45827 (local address 127.10.33.124:45567)
0927 15:13:47.036159 (+ 19903us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:47.036164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:47.036181 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:13:47.036232 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:47.036237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:47.036307 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:47.036400 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:47.039782 (+ 3382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.041172 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.043191 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.043831 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.043951 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.047949 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:47.047959 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:47.047964 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:47.048002 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:47.051200 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:47.053089 (+ 1889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:47.053095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:47.053098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:47.053284 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:47.053837 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:47.053842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:47.053846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:47.054182 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:47.057974 (+ 3792us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:47.058101 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 15:13:47.058480 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19772,"thread_start_us":19646,"threads_started":1}
I20260927 15:13:47.059912 17647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.016485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.10.33.66:60223)
0927 15:13:47.032773 (+ 16288us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:47.032792 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:47.032910 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:47.033841 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:47.033880 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:47.034780 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:47.035641 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:47.035666 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.038014 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.038019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:47.040396 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:47.040410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.041772 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.041794 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:47.042228 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.043353 (+ 1125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:47.043388 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:47.046915 (+ 3527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:47.051117 (+ 4202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:47.051127 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:47.051152 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:47.052517 (+ 1365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:47.057645 (+ 5128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:47.057650 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:47.057653 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:47.057791 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:47.058917 (+ 1126us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:47.058921 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:47.059070 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 15:13:47.059227 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 15:13:47.059498 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16167,"mutex_wait_us":1,"spinlock_wait_cycles":23936,"thread_start_us":16025,"threads_started":1}
I20260927 15:13:47.059919 17644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.015679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:60223 (local address 127.10.33.126:32781)
0927 15:13:47.028619 (+ 12940us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:47.028627 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:47.033013 (+ 4386us) server_negotiation.cc:407] Connection header received
0927 15:13:47.033080 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:47.033085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:47.033157 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:47.033246 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:47.036652 (+ 3406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.037774 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.040555 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.041273 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.041373 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.047182 (+ 5809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:47.047190 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:47.047195 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:47.047237 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:47.050940 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:47.057314 (+ 6374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:47.057320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:47.057323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:47.057508 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:47.057922 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:47.057926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:47.057930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:47.058241 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:47.058374 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:47.059356 (+ 982us) server_negotiation.cc:299] Negotiation successful
0927 15:13:47.059571 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12776,"thread_start_us":12940,"threads_started":1}
I20260927 15:13:47.059991 17640 heartbeater.cc:347] Connected to a master server at 127.10.33.124:45567
I20260927 15:13:47.060300 17640 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:47.060974 17641 heartbeater.cc:347] Connected to a master server at 127.10.33.126:32781
I20260927 15:13:47.061137 17641 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:47.061198 17640 heartbeater.cc:511] Master 127.10.33.124:45567 requested a full tablet report, sending...
I20260927 15:13:47.061580 17641 heartbeater.cc:511] Master 127.10.33.126:32781 requested a full tablet report, sending...
I20260927 15:13:47.061133 17650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.033161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:37211 (local address 127.10.33.66:45319)
0927 15:13:47.038149 (+ 4988us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:47.038173 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:47.038292 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:47.043351 (+ 5059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:47.043356 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:47.043981 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:47.044383 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:47.044397 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.045930 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.045934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:47.048224 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:47.048235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.049292 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.049300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:47.049480 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.050056 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:47.050077 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:47.052563 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:47.056254 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:47.056258 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:47.056261 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:47.056793 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:47.057992 (+ 1199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:47.057998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:47.058002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:47.058156 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:47.059020 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:47.059025 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:47.059192 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 15:13:47.059315 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 15:13:47.060723 (+ 1408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4831,"mutex_wait_us":1129,"thread_start_us":5220,"threads_started":1}
I20260927 15:13:47.062528 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.033165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:45319 (local address 127.10.33.125:37211)
0927 15:13:47.042949 (+ 9784us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:47.042956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:47.042975 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:13:47.043046 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:47.043052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:47.043135 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:47.044523 (+ 1388us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:47.044675 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.045773 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.048356 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.048932 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.049048 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.052768 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:47.052774 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:47.052779 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:47.052831 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:47.056094 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:47.056942 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:47.056948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:47.056952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:47.057807 (+ 855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:47.058276 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:47.058281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:47.058285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:47.058600 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:47.058736 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:47.059586 (+ 850us) server_negotiation.cc:299] Negotiation successful
0927 15:13:47.059798 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9656,"thread_start_us":9526,"threads_started":1}
I20260927 15:13:47.062901 16528 ts_manager.cc:194] Re-registered known tserver with Master: 49233ab4d77a4c58b34c9f40d42ff227 (127.10.33.66:37121)
I20260927 15:13:47.063094 17315 ts_manager.cc:194] Registered new tserver with Master: 49233ab4d77a4c58b34c9f40d42ff227 (127.10.33.66:37121)
I20260927 15:13:47.063344 17642 heartbeater.cc:347] Connected to a master server at 127.10.33.125:37211
I20260927 15:13:47.063459 17642 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:47.063855 17642 heartbeater.cc:511] Master 127.10.33.125:37211 requested a full tablet report, sending...
I20260927 15:13:47.063863 16528 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:60223
I20260927 15:13:47.064874 17126 ts_manager.cc:194] Registered new tserver with Master: 49233ab4d77a4c58b34c9f40d42ff227 (127.10.33.66:37121)
I20260927 15:13:47.142037 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 16856
I20260927 15:13:47.155220 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:34725
--local_ip_for_outbound_sockets=127.10.33.67
--tserver_master_addrs=127.10.33.126:32781
--webserver_port=38807
--webserver_interface=127.10.33.67
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.67.keytab
--principal=oryx/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.33.126:32781,127.10.33.125:37211
--tserver_master_addrs=127.10.33.126:32781,127.10.33.125:37211,127.10.33.124:45567 with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:13:47.437429 17495 heartbeater.cc:503] Master 127.10.33.126:32781 was elected leader, sending a full tablet report...
W20260927 15:13:47.480822 17653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:47.481117 17653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:47.481230 17653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:47.488489 17653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:47.488615 17653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:47.488766 17653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:13:47.498603 17653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.67.keytab
--principal=oryx/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:34725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=38807
--tserver_master_addrs=127.10.33.126:32781,127.10.33.125:37211,127.10.33.124:45567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:47.499888 17653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:47.501206 17653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:47.513679 17658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:47.513799 17661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:47.513770 17659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:47.514801 17653 server_base.cc:1034] running on GCE node
I20260927 15:13:47.515422 17653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:47.516327 17653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:47.517534 17653 hybrid_clock.cc:648] HybridClock initialized: now 1790522027517470 us; error 59 us; skew 500 ppm
I20260927 15:13:47.517913 17653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:47 dist-test-slave-f5h6 krb5kdc[16483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522027, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:47.521200 17653 init.cc:377] Logged in from keytab as oryx/127.10.33.67@KRBTEST.COM (short username oryx)
I20260927 15:13:47.522806 17653 webserver.cc:460] Webserver started at http://127.10.33.67:38807/ using document root <none> and password file <none>
I20260927 15:13:47.523516 17653 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:47.523644 17653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:47.529904 17653 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:13:47.533030 17668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:47.534816 17653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 15:13:47.534998 17653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "8f1dea582f4446818f93998cdd54100c"
format_stamp: "Formatted at 2026-09-27 15:13:44 on dist-test-slave-f5h6"
server_key: "a4a07f57a542449cbaa152dbe55a5f72"
server_key_iv: "5324a59cdc599757084895fa10fc4b1b"
server_key_version: "encryptionkey@0"
I20260927 15:13:47.536011 17653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:47.558843 17653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:47.563405 17653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:47.563676 17653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:47.564716 17653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:13:47.566231 17653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:47.566339 17653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:47.566449 17653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:47.566532 17653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:47.601912 17653 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:34725
I20260927 15:13:47.602003 17780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:34725 every 8 connection(s)
I20260927 15:13:47.603358 17653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:13:47.605288 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 17653
Sep 27 15:13:47 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522027, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.125@KRBTEST.COM
Sep 27 15:13:47 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522027, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM
Sep 27 15:13:47 dist-test-slave-f5h6 krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522027, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.124@KRBTEST.COM
I20260927 15:13:47.665474 17794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.618219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45392 (local address 127.10.33.126:32781)
0927 15:13:47.631620 (+ 13401us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:47.631628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:47.631654 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:13:47.631728 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:47.631735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:47.631829 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:47.631947 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:47.637630 (+ 5683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.638750 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.645476 (+ 6726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.646114 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.646255 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.655909 (+ 9654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:47.655917 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:47.655921 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:47.655964 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:47.660758 (+ 4794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:47.662642 (+ 1884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:47.662649 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:47.662653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:47.662837 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:47.663320 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:47.663325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:47.663329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:47.663726 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:47.664320 (+ 594us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:47.664798 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 15:13:47.665043 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13240,"thread_start_us":13105,"threads_started":1}
I20260927 15:13:47.668702 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.622746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45448 (local address 127.10.33.124:45567)
0927 15:13:47.639292 (+ 16546us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:47.639298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:47.639316 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:13:47.639367 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:47.639372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:47.639447 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:47.639559 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:47.643969 (+ 4410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.647549 (+ 3580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.649663 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.650315 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.650441 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.658252 (+ 7811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:47.658260 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:47.658265 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:47.658313 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:47.662288 (+ 3975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:47.665719 (+ 3431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:47.665725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:47.665728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:47.666036 (+ 308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:47.666494 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:47.666498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:47.666501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:47.666879 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:47.667099 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:47.668114 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0927 15:13:47.668346 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2147,"server-negotiator.queue_time_us":16406,"thread_start_us":13473,"threads_started":1}
I20260927 15:13:47.670375 17799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.641947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.10.33.67:54637)
0927 15:13:47.645641 (+ 3694us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:47.645664 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:47.645813 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:47.646735 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:47.646740 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:47.647245 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:47.647552 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:47.647563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.649392 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.649398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:47.651067 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:47.651080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.653034 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.653040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:47.653292 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.654754 (+ 1462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:47.654774 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:47.658024 (+ 3250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:47.665035 (+ 7011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:47.665040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:47.665044 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:47.665738 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:47.666700 (+ 962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:47.666704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:47.666707 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:47.666862 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:47.667526 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:47.667534 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:47.668004 (+ 470us) client_negotiation.cc:769] Sending connection context
0927 15:13:47.668339 (+ 335us) client_negotiation.cc:240] Negotiation successful
0927 15:13:47.668882 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3503,"thread_start_us":3558,"threads_started":1}
I20260927 15:13:47.671861 17798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.627089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44856 (local address 127.10.33.125:37211)
0927 15:13:47.643488 (+ 16399us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:47.643495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:47.643511 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:13:47.643563 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:47.643567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:47.643635 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:47.649240 (+ 5605us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:47.650122 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.651294 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.653774 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.654394 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.654514 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.659139 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:47.659146 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:47.659151 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:47.659206 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:47.667629 (+ 8423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:47.668451 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:47.668457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:47.668460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:47.668626 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:47.670082 (+ 1456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:47.670087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:47.670090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:47.670452 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:47.670685 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:47.671243 (+ 558us) server_negotiation.cc:299] Negotiation successful
0927 15:13:47.671474 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16275,"thread_start_us":16157,"threads_started":1}
I20260927 15:13:47.671895 17788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.607802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:45567 (local address 127.10.33.67:34031)
0927 15:13:47.633295 (+ 25493us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:47.633359 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:47.635836 (+ 2477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:47.636858 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:47.636874 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:47.637831 (+ 957us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:47.638738 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:47.638766 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.640328 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.640334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:47.644256 (+ 3922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:47.644271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.646543 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.646551 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:47.646861 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.652734 (+ 5873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:47.652777 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:47.658873 (+ 6096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:47.663933 (+ 5060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:47.663944 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:47.663966 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:47.664855 (+ 889us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:47.666302 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:47.666307 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:47.666310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:47.666582 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:47.670928 (+ 4346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:47.670932 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:47.671066 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 15:13:47.671196 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 15:13:47.671477 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14449,"spinlock_wait_cycles":111744,"thread_start_us":20301,"threads_started":1}
I20260927 15:13:47.672794 17795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.608592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:34031 (local address 127.10.33.124:45567)
0927 15:13:47.635275 (+ 26683us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:47.635282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:47.635307 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:13:47.636191 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:47.636197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:47.636277 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:47.636375 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:47.638962 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.640106 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.644429 (+ 4323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.645151 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.645272 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.659106 (+ 13834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:47.659112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:47.659117 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:47.659152 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:47.663763 (+ 4611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:47.666021 (+ 2258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:47.666026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:47.666029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:47.666163 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:47.666985 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:47.666990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:47.666993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:47.667319 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:47.667446 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:47.672254 (+ 4808us) server_negotiation.cc:299] Negotiation successful
0927 15:13:47.672497 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":26544,"thread_start_us":30407,"threads_started":1}
I20260927 15:13:47.673267 17800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.641982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:54637 (local address 127.10.33.126:32781)
0927 15:13:47.646295 (+ 4313us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:47.646301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:47.646316 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:13:47.646362 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:47.646367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:47.646433 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:47.647674 (+ 1241us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:47.647893 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.649066 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.651236 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.651746 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.651836 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.659058 (+ 7222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:47.659065 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:47.659069 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:47.659112 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:47.664057 (+ 4945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:47.666426 (+ 2369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:47.666432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:47.666435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:47.666573 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:47.667045 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:47.667051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:47.667054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:47.667389 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:47.667528 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:47.669599 (+ 2071us) server_negotiation.cc:299] Negotiation successful
0927 15:13:47.669826 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4188,"thread_start_us":4079,"threads_started":1}
I20260927 15:13:47.674307 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.607204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:60505 (local address 127.10.33.125:37211)
0927 15:13:47.636633 (+ 29429us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:47.636640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:47.640332 (+ 3692us) server_negotiation.cc:407] Connection header received
0927 15:13:47.640534 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:47.640541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:47.640642 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:47.640749 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:47.644021 (+ 3272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.645072 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.651282 (+ 6210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.651901 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.652001 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.661467 (+ 9466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:47.661475 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:47.661479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:47.661538 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:47.665248 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:47.666743 (+ 1495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:47.666749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:47.666752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:47.666905 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:47.672088 (+ 5183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:47.672094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:47.672097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:47.672467 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:47.672601 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:47.673625 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0927 15:13:47.673906 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29317,"thread_start_us":29215,"threads_started":1}
I20260927 15:13:47.674705 17782 heartbeater.cc:347] Connected to a master server at 127.10.33.126:32781
I20260927 15:13:47.675064 17782 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:47.675992 17782 heartbeater.cc:511] Master 127.10.33.126:32781 requested a full tablet report, sending...
I20260927 15:13:47.676371 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.607810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:37211 (local address 127.10.33.67:60505)
0927 15:13:47.640142 (+ 32332us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:47.640162 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:47.640375 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:47.640997 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:47.641002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:47.642646 (+ 1644us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:47.642921 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:47.642933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.645296 (+ 2363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.645301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:47.650826 (+ 5525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:47.650843 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.652287 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.652294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:47.652500 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.653603 (+ 1103us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:47.653626 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:47.661201 (+ 7575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:47.665729 (+ 4528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:47.665734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:47.665738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:47.666286 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:47.669150 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:47.669154 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:47.669157 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:47.669352 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:47.672688 (+ 3336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:47.672692 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:47.672854 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 15:13:47.674140 (+ 1286us) client_negotiation.cc:240] Negotiation successful
0927 15:13:47.674535 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32148,"mutex_wait_us":830,"thread_start_us":33290,"threads_started":1}
I20260927 15:13:47.676767 17781 heartbeater.cc:347] Connected to a master server at 127.10.33.124:45567
I20260927 15:13:47.676900 17781 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:47.677307 17781 heartbeater.cc:511] Master 127.10.33.124:45567 requested a full tablet report, sending...
I20260927 15:13:47.677441 17783 heartbeater.cc:347] Connected to a master server at 127.10.33.125:37211
I20260927 15:13:47.677572 17783 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:47.677927 16527 ts_manager.cc:194] Re-registered known tserver with Master: 8f1dea582f4446818f93998cdd54100c (127.10.33.67:34725)
I20260927 15:13:47.678069 17783 heartbeater.cc:511] Master 127.10.33.125:37211 requested a full tablet report, sending...
I20260927 15:13:47.678478 17315 ts_manager.cc:194] Registered new tserver with Master: 8f1dea582f4446818f93998cdd54100c (127.10.33.67:34725)
I20260927 15:13:47.679296 16527 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:54637
I20260927 15:13:47.679809 17126 ts_manager.cc:194] Registered new tserver with Master: 8f1dea582f4446818f93998cdd54100c (127.10.33.67:34725)
I20260927 15:13:47.670790 16528 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:13:47.682719 16528 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:13:47.728440 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.707077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51704 (local address 127.10.33.67:34725)
0927 15:13:47.707711 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:47.707719 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:47.719554 (+ 11835us) server_negotiation.cc:407] Connection header received
0927 15:13:47.719903 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:47.719912 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:47.720199 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:47.720383 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:47.721481 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.723104 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.725601 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.727110 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.727226 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.727438 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:47.727980 (+ 542us) server_negotiation.cc:299] Negotiation successful
0927 15:13:47.728142 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":2026,"threads_started":1}
I20260927 15:13:47.728482 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.705658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:34725 (local address 127.0.0.1:51704)
0927 15:13:47.719397 (+ 13739us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:47.719419 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:47.719702 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:47.720719 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:47.720724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:47.720751 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:47.721285 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:47.721297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.723287 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.723292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:47.725425 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:47.725439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.727597 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.727603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:47.727801 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.727834 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:13:47.727908 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 15:13:47.728057 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13508,"mutex_wait_us":3,"thread_start_us":15901,"threads_started":1}
I20260927 15:13:47.731093 17814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.706340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49174 (local address 127.10.33.65:41861)
0927 15:13:47.706916 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:47.706927 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:47.722147 (+ 15220us) server_negotiation.cc:407] Connection header received
0927 15:13:47.722439 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:47.722449 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:47.722747 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:47.722916 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:47.724583 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.726087 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.728517 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.729990 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.730093 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.730244 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:47.730667 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 15:13:47.730820 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":1222,"threads_started":1}
I20260927 15:13:47.731142 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.705822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:41861 (local address 127.0.0.1:49174)
0927 15:13:47.722023 (+ 16201us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:47.722047 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:47.722259 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:47.723154 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:47.723159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:47.723185 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:47.723615 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:47.723626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.726284 (+ 2658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.726290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:47.728307 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:47.728321 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.730376 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.730380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:47.730498 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.730519 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:13:47.730592 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 15:13:47.730733 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15921,"mutex_wait_us":60,"thread_start_us":20367,"threads_started":1}
I20260927 15:13:47.732852 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 764b747dd1a04935a621e79babac8cbf (DEFAULT_TABLE table=test-table [id=9a0859b85792448e9028f7be37f700b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:47.734222 17429 tablet_service.cc:1467] Processing CreateTablet for tablet 764b747dd1a04935a621e79babac8cbf (DEFAULT_TABLE table=test-table [id=9a0859b85792448e9028f7be37f700b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:47.734807 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764b747dd1a04935a621e79babac8cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:47.735335 17429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764b747dd1a04935a621e79babac8cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:47.736406 17817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.708687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.10.33.66:37121)
0927 15:13:47.709204 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:47.709220 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:47.724324 (+ 15104us) server_negotiation.cc:407] Connection header received
0927 15:13:47.727906 (+ 3582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:47.727916 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:47.728255 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:47.728476 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:47.730067 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.731475 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.733736 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.735064 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.735178 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.735370 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:47.735937 (+ 567us) server_negotiation.cc:299] Negotiation successful
0927 15:13:47.736110 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":252,"threads_started":1}
I20260927 15:13:47.737052 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.708148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:37121 (local address 127.0.0.1:41140)
0927 15:13:47.724192 (+ 16044us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:47.724212 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:47.724429 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:47.728926 (+ 4497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:47.728931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:47.728960 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:47.729718 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:47.729732 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.731643 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.731648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:47.733555 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:47.733571 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.735554 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.735560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:47.735737 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.735769 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 15:13:47.736597 (+ 828us) client_negotiation.cc:240] Negotiation successful
0927 15:13:47.736745 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15906,"thread_start_us":10878,"threads_started":1}
I20260927 15:13:47.741554 17575 tablet_service.cc:1467] Processing CreateTablet for tablet 764b747dd1a04935a621e79babac8cbf (DEFAULT_TABLE table=test-table [id=9a0859b85792448e9028f7be37f700b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:47.743070 17575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764b747dd1a04935a621e79babac8cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:47.747798 17818 tablet_bootstrap.cc:492] T 764b747dd1a04935a621e79babac8cbf P 8f1dea582f4446818f93998cdd54100c: Bootstrap starting.
I20260927 15:13:47.751138 17819 tablet_bootstrap.cc:492] T 764b747dd1a04935a621e79babac8cbf P 48e1d19f3efd444fa8f0ba4fea3b7da3: Bootstrap starting.
I20260927 15:13:47.751618 17818 tablet_bootstrap.cc:654] T 764b747dd1a04935a621e79babac8cbf P 8f1dea582f4446818f93998cdd54100c: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:47.753024 17818 log.cc:824] T 764b747dd1a04935a621e79babac8cbf P 8f1dea582f4446818f93998cdd54100c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:47.754761 17819 tablet_bootstrap.cc:654] T 764b747dd1a04935a621e79babac8cbf P 48e1d19f3efd444fa8f0ba4fea3b7da3: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:47.756165 17819 log.cc:824] T 764b747dd1a04935a621e79babac8cbf P 48e1d19f3efd444fa8f0ba4fea3b7da3: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:47.758100 17822 tablet_bootstrap.cc:492] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227: Bootstrap starting.
I20260927 15:13:47.759804 17819 tablet_bootstrap.cc:492] T 764b747dd1a04935a621e79babac8cbf P 48e1d19f3efd444fa8f0ba4fea3b7da3: No bootstrap required, opened a new log
I20260927 15:13:47.760102 17819 ts_tablet_manager.cc:1397] T 764b747dd1a04935a621e79babac8cbf P 48e1d19f3efd444fa8f0ba4fea3b7da3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 15:13:47.761659 17822 tablet_bootstrap.cc:654] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:47.762933 17822 log.cc:824] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:47.763938 17818 tablet_bootstrap.cc:492] T 764b747dd1a04935a621e79babac8cbf P 8f1dea582f4446818f93998cdd54100c: No bootstrap required, opened a new log
I20260927 15:13:47.764151 17818 ts_tablet_manager.cc:1397] T 764b747dd1a04935a621e79babac8cbf P 8f1dea582f4446818f93998cdd54100c: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20260927 15:13:47.768946 17822 tablet_bootstrap.cc:492] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227: No bootstrap required, opened a new log
I20260927 15:13:47.769326 17822 ts_tablet_manager.cc:1397] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227: Time spent bootstrapping tablet: real 0.011s user 0.011s sys 0.000s
I20260927 15:13:47.770222 17819 raft_consensus.cc:348] T 764b747dd1a04935a621e79babac8cbf P 48e1d19f3efd444fa8f0ba4fea3b7da3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1dea582f4446818f93998cdd54100c" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34725 } } peers { permanent_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41861 } } peers { permanent_uuid: "49233ab4d77a4c58b34c9f40d42ff227" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37121 } }
I20260927 15:13:47.770682 17819 raft_consensus.cc:374] T 764b747dd1a04935a621e79babac8cbf P 48e1d19f3efd444fa8f0ba4fea3b7da3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:47.770792 17819 raft_consensus.cc:729] T 764b747dd1a04935a621e79babac8cbf P 48e1d19f3efd444fa8f0ba4fea3b7da3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e1d19f3efd444fa8f0ba4fea3b7da3, State: Initialized, Role: FOLLOWER
I20260927 15:13:47.771435 17819 consensus_queue.cc:260] T 764b747dd1a04935a621e79babac8cbf P 48e1d19f3efd444fa8f0ba4fea3b7da3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1dea582f4446818f93998cdd54100c" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34725 } } peers { permanent_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41861 } } peers { permanent_uuid: "49233ab4d77a4c58b34c9f40d42ff227" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37121 } }
I20260927 15:13:47.771905 17818 raft_consensus.cc:348] T 764b747dd1a04935a621e79babac8cbf P 8f1dea582f4446818f93998cdd54100c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1dea582f4446818f93998cdd54100c" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34725 } } peers { permanent_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41861 } } peers { permanent_uuid: "49233ab4d77a4c58b34c9f40d42ff227" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37121 } }
I20260927 15:13:47.772337 17818 raft_consensus.cc:374] T 764b747dd1a04935a621e79babac8cbf P 8f1dea582f4446818f93998cdd54100c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:47.772431 17818 raft_consensus.cc:729] T 764b747dd1a04935a621e79babac8cbf P 8f1dea582f4446818f93998cdd54100c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1dea582f4446818f93998cdd54100c, State: Initialized, Role: FOLLOWER
I20260927 15:13:47.773005 17818 consensus_queue.cc:260] T 764b747dd1a04935a621e79babac8cbf P 8f1dea582f4446818f93998cdd54100c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1dea582f4446818f93998cdd54100c" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34725 } } peers { permanent_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41861 } } peers { permanent_uuid: "49233ab4d77a4c58b34c9f40d42ff227" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37121 } }
I20260927 15:13:47.776960 17819 ts_tablet_manager.cc:1428] T 764b747dd1a04935a621e79babac8cbf P 48e1d19f3efd444fa8f0ba4fea3b7da3: Time spent starting tablet: real 0.017s user 0.006s sys 0.008s
I20260927 15:13:47.780854 17782 heartbeater.cc:503] Master 127.10.33.126:32781 was elected leader, sending a full tablet report...
I20260927 15:13:47.781972 17818 ts_tablet_manager.cc:1428] T 764b747dd1a04935a621e79babac8cbf P 8f1dea582f4446818f93998cdd54100c: Time spent starting tablet: real 0.017s user 0.008s sys 0.009s
I20260927 15:13:47.786266 17822 raft_consensus.cc:348] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1dea582f4446818f93998cdd54100c" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34725 } } peers { permanent_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41861 } } peers { permanent_uuid: "49233ab4d77a4c58b34c9f40d42ff227" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37121 } }
I20260927 15:13:47.786772 17822 raft_consensus.cc:374] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:47.786878 17822 raft_consensus.cc:729] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49233ab4d77a4c58b34c9f40d42ff227, State: Initialized, Role: FOLLOWER
I20260927 15:13:47.787533 17822 consensus_queue.cc:260] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1dea582f4446818f93998cdd54100c" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34725 } } peers { permanent_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41861 } } peers { permanent_uuid: "49233ab4d77a4c58b34c9f40d42ff227" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37121 } }
I20260927 15:13:47.788738 17641 heartbeater.cc:503] Master 127.10.33.126:32781 was elected leader, sending a full tablet report...
I20260927 15:13:47.789322 17822 ts_tablet_manager.cc:1428] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227: Time spent starting tablet: real 0.020s user 0.006s sys 0.007s
W20260927 15:13:47.857511 17784 tablet.cc:2369] T 764b747dd1a04935a621e79babac8cbf P 8f1dea582f4446818f93998cdd54100c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:13:47.901563 17826 raft_consensus.cc:482] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:13:47.901916 17826 raft_consensus.cc:504] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1dea582f4446818f93998cdd54100c" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34725 } } peers { permanent_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41861 } } peers { permanent_uuid: "49233ab4d77a4c58b34c9f40d42ff227" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37121 } }
W20260927 15:13:47.903010 17497 tablet.cc:2369] T 764b747dd1a04935a621e79babac8cbf P 48e1d19f3efd444fa8f0ba4fea3b7da3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:13:47.903965 17826 leader_election.cc:288] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f1dea582f4446818f93998cdd54100c (127.10.33.67:34725), 48e1d19f3efd444fa8f0ba4fea3b7da3 (127.10.33.65:41861)
I20260927 15:13:47.911151 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.904385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:40401 (local address 127.10.33.67:34725)
0927 15:13:47.904586 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:47.904592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:47.905271 (+ 679us) server_negotiation.cc:407] Connection header received
0927 15:13:47.905512 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:47.905517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:47.905583 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:47.905649 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:47.906223 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.907027 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.908967 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.910142 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.910286 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.910359 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:47.910780 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 15:13:47.910916 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:13:47.911151 17814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.904480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:46361 (local address 127.10.33.65:41861)
0927 15:13:47.904690 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:47.904697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:47.905352 (+ 655us) server_negotiation.cc:407] Connection header received
0927 15:13:47.905538 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:47.905542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:47.905598 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:47.905688 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:47.906301 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.907059 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.908875 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.910086 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.910338 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.910401 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:47.910780 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 15:13:47.910911 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:13:47.911434 17827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.904175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:34725 (local address 127.10.33.66:40401)
0927 15:13:47.905163 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:47.905179 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:47.905371 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:47.905688 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:47.905693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:47.905715 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:47.906068 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:47.906080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.907206 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.907211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:47.908761 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:47.908775 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.910314 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.910320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:47.910555 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.910642 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 15:13:47.910739 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 15:13:47.910960 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":203,"thread_start_us":472,"threads_started":1}
I20260927 15:13:47.911454 17828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.904416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:41861 (local address 127.10.33.66:46361)
0927 15:13:47.905242 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:47.905260 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:47.905392 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:47.905759 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:47.905767 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:47.905812 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:47.906140 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:47.906151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.907192 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.907196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:47.908721 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:47.908733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.910293 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.910303 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:47.910504 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.910642 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 15:13:47.910739 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 15:13:47.910960 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":127,"spinlock_wait_cycles":1792,"thread_start_us":473,"threads_started":1}
I20260927 15:13:47.912477 17449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764b747dd1a04935a621e79babac8cbf" candidate_uuid: "49233ab4d77a4c58b34c9f40d42ff227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" is_pre_election: true
I20260927 15:13:47.912582 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764b747dd1a04935a621e79babac8cbf" candidate_uuid: "49233ab4d77a4c58b34c9f40d42ff227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1dea582f4446818f93998cdd54100c" is_pre_election: true
I20260927 15:13:47.912967 17449 raft_consensus.cc:2454] T 764b747dd1a04935a621e79babac8cbf P 48e1d19f3efd444fa8f0ba4fea3b7da3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49233ab4d77a4c58b34c9f40d42ff227 in term 0.
I20260927 15:13:47.913045 17736 raft_consensus.cc:2454] T 764b747dd1a04935a621e79babac8cbf P 8f1dea582f4446818f93998cdd54100c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49233ab4d77a4c58b34c9f40d42ff227 in term 0.
I20260927 15:13:47.913640 17530 leader_election.cc:302] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e1d19f3efd444fa8f0ba4fea3b7da3, 49233ab4d77a4c58b34c9f40d42ff227; no voters:
I20260927 15:13:47.914067 17826 raft_consensus.cc:2781] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:13:47.914264 17826 raft_consensus.cc:482] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:13:47.914358 17826 raft_consensus.cc:3037] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:47.916718 17826 raft_consensus.cc:504] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1dea582f4446818f93998cdd54100c" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34725 } } peers { permanent_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41861 } } peers { permanent_uuid: "49233ab4d77a4c58b34c9f40d42ff227" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37121 } }
I20260927 15:13:47.917356 17826 leader_election.cc:288] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227 [CANDIDATE]: Term 1 election: Requested vote from peers 8f1dea582f4446818f93998cdd54100c (127.10.33.67:34725), 48e1d19f3efd444fa8f0ba4fea3b7da3 (127.10.33.65:41861)
I20260927 15:13:47.917843 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764b747dd1a04935a621e79babac8cbf" candidate_uuid: "49233ab4d77a4c58b34c9f40d42ff227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1dea582f4446818f93998cdd54100c"
I20260927 15:13:47.917994 17449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764b747dd1a04935a621e79babac8cbf" candidate_uuid: "49233ab4d77a4c58b34c9f40d42ff227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3"
I20260927 15:13:47.918080 17736 raft_consensus.cc:3037] T 764b747dd1a04935a621e79babac8cbf P 8f1dea582f4446818f93998cdd54100c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:47.918226 17449 raft_consensus.cc:3037] T 764b747dd1a04935a621e79babac8cbf P 48e1d19f3efd444fa8f0ba4fea3b7da3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:47.920581 17449 raft_consensus.cc:2454] T 764b747dd1a04935a621e79babac8cbf P 48e1d19f3efd444fa8f0ba4fea3b7da3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49233ab4d77a4c58b34c9f40d42ff227 in term 1.
I20260927 15:13:47.920579 17736 raft_consensus.cc:2454] T 764b747dd1a04935a621e79babac8cbf P 8f1dea582f4446818f93998cdd54100c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49233ab4d77a4c58b34c9f40d42ff227 in term 1.
I20260927 15:13:47.921103 17528 leader_election.cc:302] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49233ab4d77a4c58b34c9f40d42ff227, 8f1dea582f4446818f93998cdd54100c; no voters:
I20260927 15:13:47.921568 17826 raft_consensus.cc:2781] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:47.922065 17826 raft_consensus.cc:686] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227 [term 1 LEADER]: Becoming Leader. State: Replica: 49233ab4d77a4c58b34c9f40d42ff227, State: Running, Role: LEADER
I20260927 15:13:47.922600 17826 consensus_queue.cc:237] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1dea582f4446818f93998cdd54100c" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34725 } } peers { permanent_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41861 } } peers { permanent_uuid: "49233ab4d77a4c58b34c9f40d42ff227" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37121 } }
I20260927 15:13:47.927814 16527 catalog_manager.cc:5495] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49233ab4d77a4c58b34c9f40d42ff227 (127.10.33.66). New cstate: current_term: 1 leader_uuid: "49233ab4d77a4c58b34c9f40d42ff227" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1dea582f4446818f93998cdd54100c" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49233ab4d77a4c58b34c9f40d42ff227" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37121 } health_report { overall_health: HEALTHY } } }
I20260927 15:13:47.975785 17817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:47.969668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41156 (local address 127.10.33.66:37121)
0927 15:13:47.970022 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:47.970029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:47.970089 (+ 60us) server_negotiation.cc:407] Connection header received
0927 15:13:47.970193 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:47.970200 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:47.970305 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:47.970450 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:13:47.970796 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.971861 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.973163 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:47.973785 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:47.973935 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:47.974375 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:13:47.975089 (+ 714us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:13:47.975230 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:47.975410 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 15:13:47.975495 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260927 15:13:47.983937 17449 raft_consensus.cc:1261] T 764b747dd1a04935a621e79babac8cbf P 48e1d19f3efd444fa8f0ba4fea3b7da3 [term 1 FOLLOWER]: Refusing update from remote peer 49233ab4d77a4c58b34c9f40d42ff227: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:47.983947 17736 raft_consensus.cc:1261] T 764b747dd1a04935a621e79babac8cbf P 8f1dea582f4446818f93998cdd54100c [term 1 FOLLOWER]: Refusing update from remote peer 49233ab4d77a4c58b34c9f40d42ff227: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:47.985216 17829 consensus_queue.cc:1035] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f1dea582f4446818f93998cdd54100c" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:47.985595 17826 consensus_queue.cc:1035] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:48.002349 17833 mvcc.cc:205] Tried to move back new op lower bound from 7333978226612473856 to 7333978226374754304. Current Snapshot: MvccSnapshot[applied={T|T < 7333978226612473856}]
I20260927 15:13:48.003369 17835 mvcc.cc:205] Tried to move back new op lower bound from 7333978226612473856 to 7333978226374754304. Current Snapshot: MvccSnapshot[applied={T|T < 7333978226612473856}]
I20260927 15:13:48.006954 17834 mvcc.cc:205] Tried to move back new op lower bound from 7333978226612473856 to 7333978226374754304. Current Snapshot: MvccSnapshot[applied={T|T < 7333978226612473856}]
I20260927 15:13:48.022912 16528 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:48.023236 16528 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:48.030351 16528 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 48af3088ca81428888ca24f47977bbe5: Sending DeleteTablet for 3 replicas of tablet 764b747dd1a04935a621e79babac8cbf
I20260927 15:13:48.031524 17429 tablet_service.cc:1514] Processing DeleteTablet for tablet 764b747dd1a04935a621e79babac8cbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:48 UTC) from {username='oryx', principal='oryx/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:49174
I20260927 15:13:48.031677 17575 tablet_service.cc:1514] Processing DeleteTablet for tablet 764b747dd1a04935a621e79babac8cbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:48 UTC) from {username='oryx', principal='oryx/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:41140
I20260927 15:13:48.032725 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 764b747dd1a04935a621e79babac8cbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:48 UTC) from {username='oryx', principal='oryx/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:51704
I20260927 15:13:48.052723 17844 tablet_replica.cc:330] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227: stopping tablet replica
I20260927 15:13:48.053169 17844 raft_consensus.cc:2229] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:13:48.053874 17844 raft_consensus.cc:2258] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:48.054804 17845 tablet_replica.cc:330] T 764b747dd1a04935a621e79babac8cbf P 8f1dea582f4446818f93998cdd54100c: stopping tablet replica
I20260927 15:13:48.055243 17845 raft_consensus.cc:2229] T 764b747dd1a04935a621e79babac8cbf P 8f1dea582f4446818f93998cdd54100c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:48.055366 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 16496
I20260927 15:13:48.055799 17845 raft_consensus.cc:2258] T 764b747dd1a04935a621e79babac8cbf P 8f1dea582f4446818f93998cdd54100c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:48.056391 16558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:48.058367 17844 ts_tablet_manager.cc:1905] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:48.059734 17843 tablet_replica.cc:330] T 764b747dd1a04935a621e79babac8cbf P 48e1d19f3efd444fa8f0ba4fea3b7da3: stopping tablet replica
I20260927 15:13:48.060168 17843 raft_consensus.cc:2229] T 764b747dd1a04935a621e79babac8cbf P 48e1d19f3efd444fa8f0ba4fea3b7da3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:48.060702 17843 raft_consensus.cc:2258] T 764b747dd1a04935a621e79babac8cbf P 48e1d19f3efd444fa8f0ba4fea3b7da3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:48.064898 17845 ts_tablet_manager.cc:1905] T 764b747dd1a04935a621e79babac8cbf P 8f1dea582f4446818f93998cdd54100c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:48.065562 17843 ts_tablet_manager.cc:1905] T 764b747dd1a04935a621e79babac8cbf P 48e1d19f3efd444fa8f0ba4fea3b7da3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:48.073355 17844 ts_tablet_manager.cc:1918] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:13:48.073575 17844 log.cc:1196] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/764b747dd1a04935a621e79babac8cbf
I20260927 15:13:48.074702 17844 ts_tablet_manager.cc:1939] T 764b747dd1a04935a621e79babac8cbf P 49233ab4d77a4c58b34c9f40d42ff227: Deleting consensus metadata
I20260927 15:13:48.076371 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:48.034568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44864 (local address 127.10.33.125:37211)
0927 15:13:48.034748 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:48.034755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:48.052734 (+ 17979us) server_negotiation.cc:407] Connection header received
0927 15:13:48.056250 (+ 3516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:48.056257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:48.056363 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:48.056441 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:48.061140 (+ 4699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:48.062316 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:48.064217 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:48.064880 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:48.064985 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:48.068576 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:48.068584 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:48.068589 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:48.068634 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:48.072344 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:48.073809 (+ 1465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:48.073816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:48.073819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:48.073996 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:48.074534 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:48.074540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:48.074543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:48.074914 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:48.075127 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:48.075751 (+ 624us) server_negotiation.cc:299] Negotiation successful
0927 15:13:48.076006 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:13:48.079012 17845 ts_tablet_manager.cc:1918] T 764b747dd1a04935a621e79babac8cbf P 8f1dea582f4446818f93998cdd54100c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:13:48.079193 17845 log.cc:1196] T 764b747dd1a04935a621e79babac8cbf P 8f1dea582f4446818f93998cdd54100c: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/764b747dd1a04935a621e79babac8cbf
I20260927 15:13:48.079684 17845 ts_tablet_manager.cc:1939] T 764b747dd1a04935a621e79babac8cbf P 8f1dea582f4446818f93998cdd54100c: Deleting consensus metadata
I20260927 15:13:48.085345 17843 ts_tablet_manager.cc:1918] T 764b747dd1a04935a621e79babac8cbf P 48e1d19f3efd444fa8f0ba4fea3b7da3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:13:48.085556 17843 log.cc:1196] T 764b747dd1a04935a621e79babac8cbf P 48e1d19f3efd444fa8f0ba4fea3b7da3: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/764b747dd1a04935a621e79babac8cbf
I20260927 15:13:48.086161 17843 ts_tablet_manager.cc:1939] T 764b747dd1a04935a621e79babac8cbf P 48e1d19f3efd444fa8f0ba4fea3b7da3: Deleting consensus metadata
I20260927 15:13:48.094846 17795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:48.054797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45458 (local address 127.10.33.124:45567)
0927 15:13:48.054981 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:48.054987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:48.078614 (+ 23627us) server_negotiation.cc:407] Connection header received
0927 15:13:48.078757 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:48.078763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:48.078848 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:48.078952 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:48.080571 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:48.081561 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:48.083413 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:48.084045 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:48.084185 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:48.088186 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:48.088192 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:48.088197 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:48.088244 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:48.091590 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:48.092469 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:48.092475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:48.092479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:48.092648 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:48.093062 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:48.093066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:48.093068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:48.093433 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:48.093781 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:48.094228 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 15:13:48.094484 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:13:48.202432 16512 catalog_manager.cc:4841] TS 8f1dea582f4446818f93998cdd54100c (127.10.33.67:34725): tablet 764b747dd1a04935a621e79babac8cbf (table test-table [id=9a0859b85792448e9028f7be37f700b4]) successfully deleted
I20260927 15:13:48.202451 16514 catalog_manager.cc:4841] TS 48e1d19f3efd444fa8f0ba4fea3b7da3 (127.10.33.65:41861): tablet 764b747dd1a04935a621e79babac8cbf (table test-table [id=9a0859b85792448e9028f7be37f700b4]) successfully deleted
I20260927 15:13:48.202886 16513 catalog_manager.cc:4841] TS 49233ab4d77a4c58b34c9f40d42ff227 (127.10.33.66:37121): tablet 764b747dd1a04935a621e79babac8cbf (table test-table [id=9a0859b85792448e9028f7be37f700b4]) successfully deleted
I20260927 15:13:48.203636 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 16496
I20260927 15:13:48.244632 17795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:48.226030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45462 (local address 127.10.33.124:45567)
0927 15:13:48.226217 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:48.226224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:48.227080 (+ 856us) server_negotiation.cc:407] Connection header received
0927 15:13:48.227457 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:48.227464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:48.227551 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:48.227625 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:48.229925 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:48.231170 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:48.233232 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:48.233925 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:48.234028 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:48.238252 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:48.238260 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:48.238265 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:48.238315 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:48.241628 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:48.242566 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:48.242572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:48.242576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:48.242717 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:48.243159 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:48.243164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:48.243167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:48.243491 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:48.243645 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:48.244094 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 15:13:48.244296 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:13:48.255887 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:48.226836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44874 (local address 127.10.33.125:37211)
0927 15:13:48.226997 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:48.227003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:48.240987 (+ 13984us) server_negotiation.cc:407] Connection header received
0927 15:13:48.241145 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:48.241151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:48.241232 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:48.241329 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:48.243403 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:48.244506 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:48.246187 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:48.246811 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:48.246932 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:48.249660 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:48.249667 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:48.249673 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:48.249718 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:48.252882 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:48.253644 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:48.253651 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:48.253655 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:48.253817 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:48.254287 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:48.254292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:48.254296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:48.254676 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:48.254926 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:48.255248 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 15:13:48.255517 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:13:49.008489 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.007695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.10.33.65:54761)
0927 15:13:49.008161 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.008281 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":253,"threads_started":1}
W20260927 15:13:49.008845 17495 heartbeater.cc:644] Failed to heartbeat to 127.10.33.126:32781 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
I20260927 15:13:49.012866 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.011809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.10.33.67:35199)
0927 15:13:49.012568 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.012678 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":568,"threads_started":1}
W20260927 15:13:49.013396 17782 heartbeater.cc:644] Failed to heartbeat to 127.10.33.126:32781 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
I20260927 15:13:49.014582 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.013939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.10.33.66:59707)
0927 15:13:49.014323 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.014424 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":187,"threads_started":1}
W20260927 15:13:49.014880 17641 heartbeater.cc:644] Failed to heartbeat to 127.10.33.126:32781 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
I20260927 15:13:49.531358 17858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48af3088ca81428888ca24f47977bbe5)
I20260927 15:13:49.531728 17858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: false } }
I20260927 15:13:49.533464 17858 leader_election.cc:288] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 48af3088ca81428888ca24f47977bbe5 (127.10.33.126:32781), 4bf082f24f3f46be9dcb1fddb0ff06c2 (127.10.33.125:37211)
I20260927 15:13:49.535071 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.533758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.0.0.1:45804)
0927 15:13:49.534433 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.534645 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":394,"threads_started":1}
W20260927 15:13:49.535375 17298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
I20260927 15:13:49.536757 17863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48af3088ca81428888ca24f47977bbe5)
I20260927 15:13:49.537434 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.536732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.0.0.1:45816)
0927 15:13:49.537188 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.537278 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346}
I20260927 15:13:49.537374 17863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: false } }
W20260927 15:13:49.537798 17298 leader_election.cc:334] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 48af3088ca81428888ca24f47977bbe5 (127.10.33.126:32781): Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
I20260927 15:13:49.539644 17863 leader_election.cc:288] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 48af3088ca81428888ca24f47977bbe5 (127.10.33.126:32781), 665386ca8bfb4d8f863d414ee55cba93 (127.10.33.124:45567)
I20260927 15:13:49.541348 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.539730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.0.0.1:45832)
0927 15:13:49.540379 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.540819 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":271,"threads_started":1}
W20260927 15:13:49.541630 17110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
I20260927 15:13:49.543879 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.534553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:37211 (local address 127.0.0.1:44880)
0927 15:13:49.535500 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.535517 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:49.535646 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:49.536101 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:49.536107 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:49.536135 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:49.536494 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:49.536507 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.538172 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.538177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:49.539545 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:49.539562 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.542998 (+ 3436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.543008 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:49.543202 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:49.543238 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:13:49.543348 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 15:13:49.543496 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":378,"threads_started":1}
I20260927 15:13:49.543915 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.534719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44880 (local address 127.10.33.125:37211)
0927 15:13:49.535570 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:49.535577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:49.535603 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:13:49.535805 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:49.535813 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:49.535909 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:49.536039 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:49.536763 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.538012 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.540640 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.542808 (+ 2168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.542960 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:49.543073 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:49.543385 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 15:13:49.543540 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":457,"threads_started":1}
I20260927 15:13:49.546837 17135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "665386ca8bfb4d8f863d414ee55cba93" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" is_pre_election: true
I20260927 15:13:49.547222 17135 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 665386ca8bfb4d8f863d414ee55cba93 in term 1.
I20260927 15:13:49.547411 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.546933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.0.0.1:45846)
0927 15:13:49.547135 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.547236 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 15:13:49.547760 17110 leader_election.cc:334] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 48af3088ca81428888ca24f47977bbe5 (127.10.33.126:32781): Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
I20260927 15:13:49.548388 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.540084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:45567 (local address 127.0.0.1:45478)
0927 15:13:49.541666 (+ 1582us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.541681 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:49.541899 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:49.542236 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:49.542241 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:49.542277 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:49.542599 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:49.542609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.544109 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.544116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:49.545668 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:49.545683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.547570 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.547577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:49.547828 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:49.547868 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 15:13:49.547953 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 15:13:49.548090 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1466,"thread_start_us":726,"threads_started":1}
I20260927 15:13:49.548550 17298 leader_election.cc:302] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4bf082f24f3f46be9dcb1fddb0ff06c2, 665386ca8bfb4d8f863d414ee55cba93; no voters: 48af3088ca81428888ca24f47977bbe5
I20260927 15:13:49.548966 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.540443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45478 (local address 127.10.33.124:45567)
0927 15:13:49.541468 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:49.541474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:49.541793 (+ 319us) server_negotiation.cc:407] Connection header received
0927 15:13:49.542045 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:49.542050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:49.542114 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:49.542191 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:49.542757 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.543911 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.545881 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.547147 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.548357 (+ 1210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:49.548479 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:49.548554 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 15:13:49.548660 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":813,"thread_start_us":520,"threads_started":1}
I20260927 15:13:49.549002 17858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 15:13:49.549111 17858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48af3088ca81428888ca24f47977bbe5)
I20260927 15:13:49.549208 17858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:13:49.549592 17325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "665386ca8bfb4d8f863d414ee55cba93" is_pre_election: true
I20260927 15:13:49.551682 17858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: false } }
I20260927 15:13:49.552182 17325 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bf082f24f3f46be9dcb1fddb0ff06c2 in current term 2: Already voted for candidate 665386ca8bfb4d8f863d414ee55cba93 in this term.
I20260927 15:13:49.552484 17858 leader_election.cc:288] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [CANDIDATE]: Term 2 election: Requested vote from peers 48af3088ca81428888ca24f47977bbe5 (127.10.33.126:32781), 4bf082f24f3f46be9dcb1fddb0ff06c2 (127.10.33.125:37211)
I20260927 15:13:49.553103 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.552671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.0.0.1:45860)
0927 15:13:49.552854 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.552957 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 15:13:49.553284 17135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "665386ca8bfb4d8f863d414ee55cba93" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2"
I20260927 15:13:49.553339 17113 leader_election.cc:302] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bf082f24f3f46be9dcb1fddb0ff06c2; no voters: 48af3088ca81428888ca24f47977bbe5, 665386ca8bfb4d8f863d414ee55cba93
I20260927 15:13:49.553501 17135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:13:49.554821 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.554458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.0.0.1:45868)
0927 15:13:49.554614 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.554697 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 15:13:49.554991 17298 leader_election.cc:334] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 48af3088ca81428888ca24f47977bbe5 (127.10.33.126:32781): Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
I20260927 15:13:49.556238 17135 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 665386ca8bfb4d8f863d414ee55cba93 in term 2.
I20260927 15:13:49.556454 17863 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:13:49.556819 17298 leader_election.cc:302] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4bf082f24f3f46be9dcb1fddb0ff06c2, 665386ca8bfb4d8f863d414ee55cba93; no voters: 48af3088ca81428888ca24f47977bbe5
I20260927 15:13:49.557137 17858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:13:49.557602 17858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [term 2 LEADER]: Becoming Leader. State: Replica: 665386ca8bfb4d8f863d414ee55cba93, State: Running, Role: LEADER
I20260927 15:13:49.558045 17858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: false } }
I20260927 15:13:49.559903 17868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 665386ca8bfb4d8f863d414ee55cba93. Latest consensus state: current_term: 2 leader_uuid: "665386ca8bfb4d8f863d414ee55cba93" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: false } } }
I20260927 15:13:49.560211 17868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:49.560839 17870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:13:49.564324 17870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:13:49.565157 17870 catalog_manager.cc:1259] Loaded cluster ID: 4d4d860505ed4bc6aeb9e0803aa6f40d
I20260927 15:13:49.565233 17870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:13:49.566401 17870 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:13:49.567154 17870 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93: Loaded TSK: 0
I20260927 15:13:49.568138 17870 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:13:49.590945 17315 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:13:49.599556 17315 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:13:49.605418 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.605017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.0.0.1:45888)
0927 15:13:49.605179 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.605284 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 15:13:49.605837 17135 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [term 2 FOLLOWER]: Refusing update from remote peer 665386ca8bfb4d8f863d414ee55cba93: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 15:13:49.606477 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.606102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.0.0.1:45896)
0927 15:13:49.606262 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.606342 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 15:13:49.606736 17298 consensus_peers.cc:487] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 -> Peer 48af3088ca81428888ca24f47977bbe5 (127.10.33.126:32781): Couldn't send request to peer 48af3088ca81428888ca24f47977bbe5. Status: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 15:13:49.606751 17868 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 15:13:49.609490 17863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 665386ca8bfb4d8f863d414ee55cba93. Latest consensus state: current_term: 2 leader_uuid: "665386ca8bfb4d8f863d414ee55cba93" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: false } } }
I20260927 15:13:49.609769 17863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:13:49.611436 17858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "665386ca8bfb4d8f863d414ee55cba93" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: false } } }
I20260927 15:13:49.611763 17858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93 [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:49.612342 17863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "665386ca8bfb4d8f863d414ee55cba93" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "48af3088ca81428888ca24f47977bbe5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 32781 } } peers { permanent_uuid: "4bf082f24f3f46be9dcb1fddb0ff06c2" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 37211 } attrs { promote: false } } peers { permanent_uuid: "665386ca8bfb4d8f863d414ee55cba93" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 45567 } attrs { promote: false } } }
I20260927 15:13:49.612636 17863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf082f24f3f46be9dcb1fddb0ff06c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:13:49.635537 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.619874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49186 (local address 127.10.33.65:41861)
0927 15:13:49.630435 (+ 10561us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:49.630439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:49.630455 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:13:49.630508 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:49.630512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:49.630572 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:49.631072 (+ 500us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:49.631213 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.631930 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.633614 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.634657 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.634790 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:49.634904 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:49.635137 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 15:13:49.635273 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10450,"thread_start_us":10380,"threads_started":1}
I20260927 15:13:49.635525 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.619726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:41861 (local address 127.0.0.1:49186)
0927 15:13:49.619901 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.619921 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:49.620026 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:49.630730 (+ 10704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:49.630735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:49.630761 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:49.630938 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:49.630950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.632072 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.632077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:49.633434 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:49.633449 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.634802 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.634808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:49.634988 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:49.635013 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:13:49.635093 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 15:13:49.635228 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 15:13:49.637040 17429 tablet_service.cc:1467] Processing CreateTablet for tablet 356c721eb1fb498e8987ec37a602c4d9 (DEFAULT_TABLE table=test-table [id=bfc8a1d526f844798e2eb6beac46f3ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:49.637502 17429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 356c721eb1fb498e8987ec37a602c4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:49.641937 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.620315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41172 (local address 127.10.33.66:37121)
0927 15:13:49.635975 (+ 15660us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:49.635982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:49.636008 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:13:49.636083 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:49.636089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:49.636168 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:49.636292 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:49.636727 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.637673 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.639961 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.640814 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.640988 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:49.641095 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:49.641380 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 15:13:49.641522 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15523,"thread_start_us":16076,"threads_started":1}
I20260927 15:13:49.642307 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.620092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:37121 (local address 127.0.0.1:41172)
0927 15:13:49.620320 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.620336 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:49.620429 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:49.636324 (+ 15895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:49.636329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:49.636374 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:49.636601 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:49.636609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.637833 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.637839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:49.639709 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:49.639725 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.641027 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.641033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:49.641223 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:49.641256 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:13:49.641339 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 15:13:49.641534 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":2}
I20260927 15:13:49.643277 17575 tablet_service.cc:1467] Processing CreateTablet for tablet 356c721eb1fb498e8987ec37a602c4d9 (DEFAULT_TABLE table=test-table [id=bfc8a1d526f844798e2eb6beac46f3ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:49.643702 17575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 356c721eb1fb498e8987ec37a602c4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:49.643944 17881 tablet_bootstrap.cc:492] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3: Bootstrap starting.
I20260927 15:13:49.645725 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.621020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.10.33.67:34725)
0927 15:13:49.637390 (+ 16370us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:49.637396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:49.637806 (+ 410us) server_negotiation.cc:407] Connection header received
0927 15:13:49.637896 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:49.637901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:49.637986 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:49.638085 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:49.638737 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.639760 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.643511 (+ 3751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.644702 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.644890 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:49.644995 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:49.645320 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 15:13:49.645456 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16250,"thread_start_us":16681,"threads_started":1}
I20260927 15:13:49.645767 17881 tablet_bootstrap.cc:654] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:49.646116 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.620596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:34725 (local address 127.0.0.1:51706)
0927 15:13:49.629986 (+ 9390us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.630006 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:49.630171 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:49.638112 (+ 7941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:49.638121 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:49.638147 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:49.638567 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:49.638578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.639937 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.639943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:49.643312 (+ 3369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:49.643329 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.644966 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.644973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:49.645156 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:49.645191 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 15:13:49.645289 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 15:13:49.645431 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9229,"mutex_wait_us":63,"thread_start_us":9026,"threads_started":1}
I20260927 15:13:49.646879 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 356c721eb1fb498e8987ec37a602c4d9 (DEFAULT_TABLE table=test-table [id=bfc8a1d526f844798e2eb6beac46f3ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:49.647310 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 356c721eb1fb498e8987ec37a602c4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:49.648692 17881 tablet_bootstrap.cc:492] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3: No bootstrap required, opened a new log
I20260927 15:13:49.648821 17881 ts_tablet_manager.cc:1397] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 15:13:49.649403 17881 raft_consensus.cc:348] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41861 } } peers { permanent_uuid: "49233ab4d77a4c58b34c9f40d42ff227" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37121 } } peers { permanent_uuid: "8f1dea582f4446818f93998cdd54100c" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34725 } }
I20260927 15:13:49.649559 17881 raft_consensus.cc:374] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:49.649611 17881 raft_consensus.cc:729] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e1d19f3efd444fa8f0ba4fea3b7da3, State: Initialized, Role: FOLLOWER
I20260927 15:13:49.649816 17881 consensus_queue.cc:260] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41861 } } peers { permanent_uuid: "49233ab4d77a4c58b34c9f40d42ff227" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37121 } } peers { permanent_uuid: "8f1dea582f4446818f93998cdd54100c" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34725 } }
I20260927 15:13:49.651027 17881 ts_tablet_manager.cc:1428] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:13:49.651943 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.651011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.10.33.65:33625)
0927 15:13:49.651512 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.651688 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":635,"threads_started":1}
I20260927 15:13:49.653968 17494 heartbeater.cc:503] Master 127.10.33.124:45567 was elected leader, sending a full tablet report...
I20260927 15:13:49.654680 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.654305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.10.33.65:35341)
0927 15:13:49.654450 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.654539 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 15:13:49.658357 17883 tablet_bootstrap.cc:492] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227: Bootstrap starting.
I20260927 15:13:49.659994 17886 tablet_bootstrap.cc:492] T 356c721eb1fb498e8987ec37a602c4d9 P 8f1dea582f4446818f93998cdd54100c: Bootstrap starting.
I20260927 15:13:49.660243 17883 tablet_bootstrap.cc:654] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:49.661768 17886 tablet_bootstrap.cc:654] T 356c721eb1fb498e8987ec37a602c4d9 P 8f1dea582f4446818f93998cdd54100c: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:49.662473 17883 tablet_bootstrap.cc:492] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227: No bootstrap required, opened a new log
I20260927 15:13:49.662613 17883 ts_tablet_manager.cc:1397] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260927 15:13:49.663247 17883 raft_consensus.cc:348] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41861 } } peers { permanent_uuid: "49233ab4d77a4c58b34c9f40d42ff227" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37121 } } peers { permanent_uuid: "8f1dea582f4446818f93998cdd54100c" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34725 } }
I20260927 15:13:49.663431 17883 raft_consensus.cc:374] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:49.663491 17886 tablet_bootstrap.cc:492] T 356c721eb1fb498e8987ec37a602c4d9 P 8f1dea582f4446818f93998cdd54100c: No bootstrap required, opened a new log
I20260927 15:13:49.663519 17883 raft_consensus.cc:729] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49233ab4d77a4c58b34c9f40d42ff227, State: Initialized, Role: FOLLOWER
I20260927 15:13:49.663630 17886 ts_tablet_manager.cc:1397] T 356c721eb1fb498e8987ec37a602c4d9 P 8f1dea582f4446818f93998cdd54100c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 15:13:49.663719 17883 consensus_queue.cc:260] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41861 } } peers { permanent_uuid: "49233ab4d77a4c58b34c9f40d42ff227" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37121 } } peers { permanent_uuid: "8f1dea582f4446818f93998cdd54100c" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34725 } }
I20260927 15:13:49.664382 17883 ts_tablet_manager.cc:1428] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 15:13:49.664875 17886 raft_consensus.cc:348] T 356c721eb1fb498e8987ec37a602c4d9 P 8f1dea582f4446818f93998cdd54100c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41861 } } peers { permanent_uuid: "49233ab4d77a4c58b34c9f40d42ff227" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37121 } } peers { permanent_uuid: "8f1dea582f4446818f93998cdd54100c" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34725 } }
I20260927 15:13:49.665165 17886 raft_consensus.cc:374] T 356c721eb1fb498e8987ec37a602c4d9 P 8f1dea582f4446818f93998cdd54100c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:49.665261 17886 raft_consensus.cc:729] T 356c721eb1fb498e8987ec37a602c4d9 P 8f1dea582f4446818f93998cdd54100c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1dea582f4446818f93998cdd54100c, State: Initialized, Role: FOLLOWER
I20260927 15:13:49.665537 17886 consensus_queue.cc:260] T 356c721eb1fb498e8987ec37a602c4d9 P 8f1dea582f4446818f93998cdd54100c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41861 } } peers { permanent_uuid: "49233ab4d77a4c58b34c9f40d42ff227" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37121 } } peers { permanent_uuid: "8f1dea582f4446818f93998cdd54100c" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34725 } }
I20260927 15:13:49.666257 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.664748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.10.33.66:39717)
0927 15:13:49.665622 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.665827 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":624,"threads_started":1}
I20260927 15:13:49.666440 17640 heartbeater.cc:503] Master 127.10.33.124:45567 was elected leader, sending a full tablet report...
I20260927 15:13:49.668103 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.667230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.10.33.66:43371)
0927 15:13:49.667823 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.667921 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"threads_started":1}
I20260927 15:13:49.670418 17886 ts_tablet_manager.cc:1428] T 356c721eb1fb498e8987ec37a602c4d9 P 8f1dea582f4446818f93998cdd54100c: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
I20260927 15:13:49.673444 17781 heartbeater.cc:503] Master 127.10.33.124:45567 was elected leader, sending a full tablet report...
I20260927 15:13:49.673687 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.670765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.10.33.67:42759)
0927 15:13:49.673200 (+ 2435us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.673371 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2288,"thread_start_us":2174,"threads_started":1}
I20260927 15:13:49.675243 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.674324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.10.33.67:59031)
0927 15:13:49.674849 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.674999 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":265,"threads_started":1}
I20260927 15:13:49.726943 17889 raft_consensus.cc:482] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:13:49.727137 17889 raft_consensus.cc:504] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41861 } } peers { permanent_uuid: "49233ab4d77a4c58b34c9f40d42ff227" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37121 } } peers { permanent_uuid: "8f1dea582f4446818f93998cdd54100c" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34725 } }
I20260927 15:13:49.727808 17889 leader_election.cc:288] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48e1d19f3efd444fa8f0ba4fea3b7da3 (127.10.33.65:41861), 8f1dea582f4446818f93998cdd54100c (127.10.33.67:34725)
I20260927 15:13:49.728426 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356c721eb1fb498e8987ec37a602c4d9" candidate_uuid: "49233ab4d77a4c58b34c9f40d42ff227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1dea582f4446818f93998cdd54100c" is_pre_election: true
I20260927 15:13:49.728555 17449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356c721eb1fb498e8987ec37a602c4d9" candidate_uuid: "49233ab4d77a4c58b34c9f40d42ff227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" is_pre_election: true
I20260927 15:13:49.728668 17736 raft_consensus.cc:2454] T 356c721eb1fb498e8987ec37a602c4d9 P 8f1dea582f4446818f93998cdd54100c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49233ab4d77a4c58b34c9f40d42ff227 in term 0.
I20260927 15:13:49.728821 17449 raft_consensus.cc:2454] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49233ab4d77a4c58b34c9f40d42ff227 in term 0.
I20260927 15:13:49.729293 17528 leader_election.cc:302] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 48e1d19f3efd444fa8f0ba4fea3b7da3, 49233ab4d77a4c58b34c9f40d42ff227, 8f1dea582f4446818f93998cdd54100c; no voters:
I20260927 15:13:49.729601 17889 raft_consensus.cc:2781] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:13:49.729710 17889 raft_consensus.cc:482] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:13:49.729794 17889 raft_consensus.cc:3037] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:49.731863 17889 raft_consensus.cc:504] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41861 } } peers { permanent_uuid: "49233ab4d77a4c58b34c9f40d42ff227" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37121 } } peers { permanent_uuid: "8f1dea582f4446818f93998cdd54100c" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34725 } }
I20260927 15:13:49.732391 17889 leader_election.cc:288] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227 [CANDIDATE]: Term 1 election: Requested vote from peers 48e1d19f3efd444fa8f0ba4fea3b7da3 (127.10.33.65:41861), 8f1dea582f4446818f93998cdd54100c (127.10.33.67:34725)
I20260927 15:13:49.732899 17449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356c721eb1fb498e8987ec37a602c4d9" candidate_uuid: "49233ab4d77a4c58b34c9f40d42ff227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3"
I20260927 15:13:49.733040 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356c721eb1fb498e8987ec37a602c4d9" candidate_uuid: "49233ab4d77a4c58b34c9f40d42ff227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1dea582f4446818f93998cdd54100c"
I20260927 15:13:49.733121 17449 raft_consensus.cc:3037] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:49.733250 17736 raft_consensus.cc:3037] T 356c721eb1fb498e8987ec37a602c4d9 P 8f1dea582f4446818f93998cdd54100c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:49.736119 17449 raft_consensus.cc:2454] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49233ab4d77a4c58b34c9f40d42ff227 in term 1.
I20260927 15:13:49.736119 17736 raft_consensus.cc:2454] T 356c721eb1fb498e8987ec37a602c4d9 P 8f1dea582f4446818f93998cdd54100c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49233ab4d77a4c58b34c9f40d42ff227 in term 1.
I20260927 15:13:49.736590 17530 leader_election.cc:302] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e1d19f3efd444fa8f0ba4fea3b7da3, 49233ab4d77a4c58b34c9f40d42ff227; no voters:
I20260927 15:13:49.736933 17889 raft_consensus.cc:2781] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:49.737409 17889 raft_consensus.cc:686] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227 [term 1 LEADER]: Becoming Leader. State: Replica: 49233ab4d77a4c58b34c9f40d42ff227, State: Running, Role: LEADER
I20260927 15:13:49.737715 17889 consensus_queue.cc:237] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41861 } } peers { permanent_uuid: "49233ab4d77a4c58b34c9f40d42ff227" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37121 } } peers { permanent_uuid: "8f1dea582f4446818f93998cdd54100c" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34725 } }
I20260927 15:13:49.739207 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.738314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.10.33.66:59303)
0927 15:13:49.738744 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.738932 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258}
I20260927 15:13:49.741084 17315 catalog_manager.cc:5495] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49233ab4d77a4c58b34c9f40d42ff227 (127.10.33.66). New cstate: current_term: 1 leader_uuid: "49233ab4d77a4c58b34c9f40d42ff227" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49233ab4d77a4c58b34c9f40d42ff227" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f1dea582f4446818f93998cdd54100c" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34725 } health_report { overall_health: UNKNOWN } } }
I20260927 15:13:49.767042 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.760857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41180 (local address 127.10.33.66:37121)
0927 15:13:49.761172 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:49.761179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:49.761272 (+ 93us) server_negotiation.cc:407] Connection header received
0927 15:13:49.761368 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:49.761373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:49.761464 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:49.761573 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:13:49.761973 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.763051 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.765019 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.765664 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.765793 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:49.766317 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:13:49.766454 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:13:49.766548 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:49.766658 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 15:13:49.766745 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
W20260927 15:13:49.768517 17643 tablet.cc:2369] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:13:49.770350 17736 raft_consensus.cc:1261] T 356c721eb1fb498e8987ec37a602c4d9 P 8f1dea582f4446818f93998cdd54100c [term 1 FOLLOWER]: Refusing update from remote peer 49233ab4d77a4c58b34c9f40d42ff227: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:49.770498 17449 raft_consensus.cc:1261] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3 [term 1 FOLLOWER]: Refusing update from remote peer 49233ab4d77a4c58b34c9f40d42ff227: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:49.770983 17895 consensus_queue.cc:1035] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f1dea582f4446818f93998cdd54100c" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:49.771314 17889 consensus_queue.cc:1035] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e1d19f3efd444fa8f0ba4fea3b7da3" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 41861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:49.773885 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.773328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.10.33.65:42613)
0927 15:13:49.773545 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.773652 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260927 15:13:49.773846 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.773452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.10.33.67:37151)
0927 15:13:49.773593 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.773679 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 15:13:49.777539 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.777110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.10.33.67:42427)
0927 15:13:49.777253 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.777356 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 15:13:49.784291 17315 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:49.784598 17315 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:49.786010 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.785592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.10.33.66:58769)
0927 15:13:49.785758 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:49.785832 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 15:13:49.790764 17315 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 665386ca8bfb4d8f863d414ee55cba93: Sending DeleteTablet for 3 replicas of tablet 356c721eb1fb498e8987ec37a602c4d9
I20260927 15:13:49.791569 17429 tablet_service.cc:1514] Processing DeleteTablet for tablet 356c721eb1fb498e8987ec37a602c4d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:49 UTC) from {username='oryx', principal='oryx/127.10.33.124@KRBTEST.COM'} at 127.0.0.1:49186
I20260927 15:13:49.791765 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 356c721eb1fb498e8987ec37a602c4d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:49 UTC) from {username='oryx', principal='oryx/127.10.33.124@KRBTEST.COM'} at 127.0.0.1:51706
I20260927 15:13:49.792001 17575 tablet_service.cc:1514] Processing DeleteTablet for tablet 356c721eb1fb498e8987ec37a602c4d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:49 UTC) from {username='oryx', principal='oryx/127.10.33.124@KRBTEST.COM'} at 127.0.0.1:41172
I20260927 15:13:49.792117 17574 tablet_service.cc:1514] Processing DeleteTablet for tablet 356c721eb1fb498e8987ec37a602c4d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:49 UTC) from {username='oryx', principal='oryx/127.10.33.124@KRBTEST.COM'} at 127.0.0.1:41172
I20260927 15:13:49.793967 17908 tablet_replica.cc:330] T 356c721eb1fb498e8987ec37a602c4d9 P 8f1dea582f4446818f93998cdd54100c: stopping tablet replica
I20260927 15:13:49.794174 17908 raft_consensus.cc:2229] T 356c721eb1fb498e8987ec37a602c4d9 P 8f1dea582f4446818f93998cdd54100c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:49.794271 17909 tablet_replica.cc:330] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227: stopping tablet replica
I20260927 15:13:49.794463 17908 raft_consensus.cc:2258] T 356c721eb1fb498e8987ec37a602c4d9 P 8f1dea582f4446818f93998cdd54100c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:49.794431 17909 raft_consensus.cc:2229] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:13:49.794728 17907 tablet_replica.cc:330] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3: stopping tablet replica
I20260927 15:13:49.794911 17907 raft_consensus.cc:2229] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:49.794909 17909 raft_consensus.cc:2258] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:49.795104 17907 pending_rounds.cc:62] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3: Trying to abort 2 pending ops.
I20260927 15:13:49.795518 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 17302
I20260927 15:13:49.796141 17909 ts_tablet_manager.cc:1905] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:49.795203 17907 pending_rounds.cc:68] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333978233933746176 op_type: WRITE_OP write_request { tablet_id: "356c721eb1fb498e8987ec37a602c4d9" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\331\347\344\325\006\"8\n\ntest-admin\022*\n bfc8a1d526f844798e2eb6beac46f3ba\020\001\030\001 \001(\001" signature: "<redacted>""\r\321\310\337n\302O\306\345i\210H3\273=6\317v\270b1\311/e\374@\322\252N\370\364\013\230\201\257\017\177\303\360\215\375\305;\263\310z\2375 P\246\360-\306#1Z\014\034u\346O\362\036" signing_key_seq_num: 0 } } request_id { client_id: "b046837f346c4440b736692303ff5f22" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260927 15:13:49.798961 17907 pending_rounds.cc:68] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333978233808347136 op_type: NO_OP noop_request { }
I20260927 15:13:49.799130 17907 raft_consensus.cc:2866] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260927 15:13:49.799278 17907 raft_consensus.cc:2258] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:49.800168 17908 ts_tablet_manager.cc:1905] T 356c721eb1fb498e8987ec37a602c4d9 P 8f1dea582f4446818f93998cdd54100c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:49.803189 17909 ts_tablet_manager.cc:1918] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:13:49.803355 17909 log.cc:1196] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/356c721eb1fb498e8987ec37a602c4d9
I20260927 15:13:49.803627 17907 ts_tablet_manager.cc:1905] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:49.803766 17909 ts_tablet_manager.cc:1939] T 356c721eb1fb498e8987ec37a602c4d9 P 49233ab4d77a4c58b34c9f40d42ff227: Deleting consensus metadata
I20260927 15:13:49.805119 17299 catalog_manager.cc:4841] TS 49233ab4d77a4c58b34c9f40d42ff227 (127.10.33.66:37121): tablet 356c721eb1fb498e8987ec37a602c4d9 (table test-table [id=bfc8a1d526f844798e2eb6beac46f3ba]) successfully deleted
W20260927 15:13:49.806205 17299 catalog_manager.cc:4808] TS 49233ab4d77a4c58b34c9f40d42ff227 (127.10.33.66:37121): delete failed for tablet 356c721eb1fb498e8987ec37a602c4d9 because the tablet was not found. No further retry: Not found: Tablet not found: 356c721eb1fb498e8987ec37a602c4d9
I20260927 15:13:49.806705 17908 ts_tablet_manager.cc:1918] T 356c721eb1fb498e8987ec37a602c4d9 P 8f1dea582f4446818f93998cdd54100c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:13:49.806854 17908 log.cc:1196] T 356c721eb1fb498e8987ec37a602c4d9 P 8f1dea582f4446818f93998cdd54100c: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/356c721eb1fb498e8987ec37a602c4d9
I20260927 15:13:49.807356 17908 ts_tablet_manager.cc:1939] T 356c721eb1fb498e8987ec37a602c4d9 P 8f1dea582f4446818f93998cdd54100c: Deleting consensus metadata
I20260927 15:13:49.808403 17298 catalog_manager.cc:4841] TS 8f1dea582f4446818f93998cdd54100c (127.10.33.67:34725): tablet 356c721eb1fb498e8987ec37a602c4d9 (table test-table [id=bfc8a1d526f844798e2eb6beac46f3ba]) successfully deleted
I20260927 15:13:49.809293 17907 ts_tablet_manager.cc:1918] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:13:49.809429 17907 log.cc:1196] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/356c721eb1fb498e8987ec37a602c4d9
I20260927 15:13:49.809828 17907 ts_tablet_manager.cc:1939] T 356c721eb1fb498e8987ec37a602c4d9 P 48e1d19f3efd444fa8f0ba4fea3b7da3: Deleting consensus metadata
I20260927 15:13:49.811179 17300 catalog_manager.cc:4841] TS 48e1d19f3efd444fa8f0ba4fea3b7da3 (127.10.33.65:41861): tablet 356c721eb1fb498e8987ec37a602c4d9 (table test-table [id=bfc8a1d526f844798e2eb6beac46f3ba]) successfully deleted
I20260927 15:13:49.815636 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.796335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49194 (local address 127.10.33.65:41861)
0927 15:13:49.796538 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:49.796544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:49.798306 (+ 1762us) server_negotiation.cc:407] Connection header received
0927 15:13:49.798795 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:49.798801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:49.798885 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:49.798978 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:49.800619 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.801329 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.802759 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.803406 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.803527 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:49.809559 (+ 6032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:49.809577 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:49.809593 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:49.809634 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:49.812983 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:49.813694 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:49.813703 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:49.813706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:49.813844 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:49.814297 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:49.814308 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:49.814310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:49.814669 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:49.814885 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:49.815127 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 15:13:49.815337 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:13:49.816232 17489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:49.908532 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 17302
I20260927 15:13:49.923322 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 17506
I20260927 15:13:49.936723 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:49.924219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41190 (local address 127.10.33.66:37121)
0927 15:13:49.924405 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:49.924411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:49.924522 (+ 111us) server_negotiation.cc:407] Connection header received
0927 15:13:49.924597 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:49.924601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:49.924699 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:49.924834 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:49.926229 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.926949 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.928115 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:49.928509 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:49.928623 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:49.931067 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:49.931082 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:49.931098 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:49.931134 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:49.934097 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:49.934794 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:49.934805 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:49.934808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:49.934940 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:49.935276 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:49.935279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:49.935281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:49.935627 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:49.935906 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:49.936200 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 15:13:49.936421 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:13:49.937384 17635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:50.057523 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 17506
I20260927 15:13:50.075496 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 17653
I20260927 15:13:50.089180 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:50.076260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.10.33.67:34725)
0927 15:13:50.076429 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:50.076435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:50.076518 (+ 83us) server_negotiation.cc:407] Connection header received
0927 15:13:50.076714 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:50.076718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:50.076804 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:50.076895 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:50.078207 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:50.078899 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:50.080057 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:50.080464 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:50.080550 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:50.083256 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:50.083272 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:50.083288 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:50.083330 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:50.086393 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:50.087248 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:50.087257 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:50.087260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:50.087401 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:50.087799 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:50.087804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:50.087807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:50.088196 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:50.088451 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:50.088706 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 15:13:50.088881 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:13:50.089950 17776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:50.149003 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:50.148357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.0.0.1:45900)
0927 15:13:50.148742 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:50.148851 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":201,"threads_started":1}
I20260927 15:13:50.150162 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:50.149843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:32781 (local address 127.0.0.1:45904)
0927 15:13:50.149993 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:50.150061 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 15:13:50.186267 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 17653
I20260927 15:13:50.201310 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 16989
I20260927 15:13:50.202462 17156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:50.319446 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 16989
I20260927 15:13:50.336892 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 17086
I20260927 15:13:50.337927 17345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:50.458582 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 17086
I20260927 15:13:50.458640 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:50.458188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:37211 (local address 127.0.0.1:44890)
0927 15:13:50.458392 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:50.458481 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:37211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
2026-09-27T15:13:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9468 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:13:50 dist-test-slave-f5h6 krb5kdc[17920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:13:50 dist-test-slave-f5h6 krb5kdc[17920](info): set up 2 sockets
Sep 27 15:13:50 dist-test-slave-f5h6 krb5kdc[17920](info): commencing operation
krb5kdc: starting...
W20260927 15:13:51.490583 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:13:51 dist-test-slave-f5h6 krb5kdc[17920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:13:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:13:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.126.keytab.
Entry for principal oryx/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.126.keytab.
I20260927 15:13:51.718683 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:33393
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.126.keytab
--principal=oryx/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:33393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:52.056378 17933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:52.056768 17933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:52.056905 17933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:52.065744 17933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:13:52.065943 17933 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 15:13:52.066023 17933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:52.066119 17933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:52.066190 17933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:13:52.066283 17933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:13:52.077136 17933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:33393
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.126.keytab
--principal=oryx/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:33393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:52.078706 17933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:52.080060 17933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:52.093051 17938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:52.093021 17941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:52.093081 17939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:52.094203 17933 server_base.cc:1034] running on GCE node
I20260927 15:13:52.095132 17933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:52.096724 17933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:52.098014 17933 hybrid_clock.cc:648] HybridClock initialized: now 1790522032097939 us; error 55 us; skew 500 ppm
I20260927 15:13:52.098366 17933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:52 dist-test-slave-f5h6 krb5kdc[17920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522032, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:52.101684 17933 init.cc:377] Logged in from keytab as oryx/127.10.33.126@KRBTEST.COM (short username oryx)
I20260927 15:13:52.103878 17933 webserver.cc:460] Webserver started at http://127.10.33.126:33873/ using document root <none> and password file <none>
I20260927 15:13:52.105286 17933 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:52.105468 17933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:52.106489 17933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:52.109941 17933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "1e07a2f8bd2a4e309e5d5c352810526a"
format_stamp: "Formatted at 2026-09-27 15:13:52 on dist-test-slave-f5h6"
server_key: "28dcadeca10490ae982bfb0abfb40de5"
server_key_iv: "7b45594cccefc0460767df9dbe2d3642"
server_key_version: "encryptionkey@0"
I20260927 15:13:52.110844 17933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "1e07a2f8bd2a4e309e5d5c352810526a"
format_stamp: "Formatted at 2026-09-27 15:13:52 on dist-test-slave-f5h6"
server_key: "28dcadeca10490ae982bfb0abfb40de5"
server_key_iv: "7b45594cccefc0460767df9dbe2d3642"
server_key_version: "encryptionkey@0"
I20260927 15:13:52.116287 17933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:13:52.120111 17948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:52.121582 17933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:13:52.121767 17933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "1e07a2f8bd2a4e309e5d5c352810526a"
format_stamp: "Formatted at 2026-09-27 15:13:52 on dist-test-slave-f5h6"
server_key: "28dcadeca10490ae982bfb0abfb40de5"
server_key_iv: "7b45594cccefc0460767df9dbe2d3642"
server_key_version: "encryptionkey@0"
I20260927 15:13:52.122006 17933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:52.144634 17933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:52.149356 17933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:52.149614 17933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:52.169663 17933 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:33393
I20260927 15:13:52.169670 18009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:33393 every 8 connection(s)
I20260927 15:13:52.170979 17933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:13:52.172338 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 17933
I20260927 15:13:52.172534 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:13:52.173362 10373 external_mini_cluster.cc:1428] Setting key 02f687c68b2eba84b201d120959e27cf
I20260927 15:13:52.175520 18010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 15:13:52 dist-test-slave-f5h6 krb5kdc[17920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM
I20260927 15:13:52.189313 18010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a: Bootstrap starting.
I20260927 15:13:52.192939 18010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:52.194132 18010 log.cc:824] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:52.197505 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:52.175318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37614 (local address 127.10.33.126:33393)
0927 15:13:52.176029 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:52.176041 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:52.176082 (+ 41us) server_negotiation.cc:407] Connection header received
0927 15:13:52.177686 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:52.177708 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:52.178143 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:52.178485 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:52.180327 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:52.181900 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:52.183683 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:52.184651 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:52.184825 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:52.189556 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:52.189566 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:52.189584 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:52.189633 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:52.192899 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:52.193694 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:52.193702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:52.193710 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:52.193947 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:52.194400 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:52.194404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:52.194406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:52.195160 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:52.195489 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:52.195974 (+ 485us) server_negotiation.cc:299] Negotiation successful
0927 15:13:52.196445 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":312,"threads_started":1}
I20260927 15:13:52.197548 18010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a: No bootstrap required, opened a new log
I20260927 15:13:52.205008 18010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e07a2f8bd2a4e309e5d5c352810526a" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 33393 } }
I20260927 15:13:52.205418 18010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:52.205513 18010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e07a2f8bd2a4e309e5d5c352810526a, State: Initialized, Role: FOLLOWER
I20260927 15:13:52.206125 18010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e07a2f8bd2a4e309e5d5c352810526a" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 33393 } }
I20260927 15:13:52.206343 18010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:13:52.206462 18010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:13:52.206619 18010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:52.208716 18010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e07a2f8bd2a4e309e5d5c352810526a" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 33393 } }
I20260927 15:13:52.209290 18010 leader_election.cc:302] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e07a2f8bd2a4e309e5d5c352810526a; no voters:
I20260927 15:13:52.209894 18010 leader_election.cc:288] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:13:52.210059 18015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:52.210743 18015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a [term 1 LEADER]: Becoming Leader. State: Replica: 1e07a2f8bd2a4e309e5d5c352810526a, State: Running, Role: LEADER
I20260927 15:13:52.211313 18015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e07a2f8bd2a4e309e5d5c352810526a" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 33393 } }
I20260927 15:13:52.211895 18010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:13:52.214485 18016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e07a2f8bd2a4e309e5d5c352810526a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e07a2f8bd2a4e309e5d5c352810526a" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 33393 } } }
I20260927 15:13:52.214425 18017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e07a2f8bd2a4e309e5d5c352810526a. Latest consensus state: current_term: 1 leader_uuid: "1e07a2f8bd2a4e309e5d5c352810526a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e07a2f8bd2a4e309e5d5c352810526a" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 33393 } } }
I20260927 15:13:52.214908 18016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:52.215071 18017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a [sys.catalog]: This master's current role is: LEADER
I20260927 15:13:52.217593 18024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:13:52.223124 18024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:13:52.231909 18024 catalog_manager.cc:1347] Generated new cluster ID: 27df1ba417234566bba9236bb710dc4e
I20260927 15:13:52.232038 18024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:13:52.248438 18024 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:13:52.250362 18024 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:13:52.260170 18024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a: Generated new TSK 0
I20260927 15:13:52.261309 18024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.33.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.33.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.33.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.65.keytab.
Entry for principal oryx/127.10.33.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.65.keytab.
I20260927 15:13:52.314226 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:33393
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.65.keytab
--principal=oryx/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:52.643479 18036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:52.643815 18036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:52.644044 18036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:52.652136 18036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:52.652312 18036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:52.652426 18036 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:13:52.652563 18036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:13:52.663923 18036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.65.keytab
--principal=oryx/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.33.126:33393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:52.665287 18036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:52.667022 18036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:52.681813 18041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:52.681878 18044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:52.681907 18042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:52.683310 18036 server_base.cc:1034] running on GCE node
I20260927 15:13:52.684017 18036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:52.685096 18036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:52.686331 18036 hybrid_clock.cc:648] HybridClock initialized: now 1790522032686284 us; error 67 us; skew 500 ppm
I20260927 15:13:52.686719 18036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:52 dist-test-slave-f5h6 krb5kdc[17920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522032, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:52.693377 18036 init.cc:377] Logged in from keytab as oryx/127.10.33.65@KRBTEST.COM (short username oryx)
I20260927 15:13:52.695015 18036 webserver.cc:460] Webserver started at http://127.10.33.65:42233/ using document root <none> and password file <none>
I20260927 15:13:52.695729 18036 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:52.695843 18036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:52.696165 18036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:52.699446 18036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "82dac9ab7a51481a9bd36d99cac38b25"
format_stamp: "Formatted at 2026-09-27 15:13:52 on dist-test-slave-f5h6"
server_key: "dcb89e8d30798eab1d1d3e9203b61ac4"
server_key_iv: "bc5fc729cf761477f86ce2593034ac99"
server_key_version: "encryptionkey@0"
I20260927 15:13:52.700243 18036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "82dac9ab7a51481a9bd36d99cac38b25"
format_stamp: "Formatted at 2026-09-27 15:13:52 on dist-test-slave-f5h6"
server_key: "dcb89e8d30798eab1d1d3e9203b61ac4"
server_key_iv: "bc5fc729cf761477f86ce2593034ac99"
server_key_version: "encryptionkey@0"
I20260927 15:13:52.705500 18036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 15:13:52.709055 18051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:52.710495 18036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 15:13:52.710645 18036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "82dac9ab7a51481a9bd36d99cac38b25"
format_stamp: "Formatted at 2026-09-27 15:13:52 on dist-test-slave-f5h6"
server_key: "dcb89e8d30798eab1d1d3e9203b61ac4"
server_key_iv: "bc5fc729cf761477f86ce2593034ac99"
server_key_version: "encryptionkey@0"
I20260927 15:13:52.710843 18036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:52.733553 18036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:52.738657 18036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:52.738925 18036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:52.741986 18036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:52.742100 18036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 15:13:52.742231 18036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:52.742300 18036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:13:52 dist-test-slave-f5h6 krb5kdc[17920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522032, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM
I20260927 15:13:52.769698 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:52.742545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:51913 (local address 127.10.33.126:33393)
0927 15:13:52.743218 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:52.743223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:52.745218 (+ 1995us) server_negotiation.cc:407] Connection header received
0927 15:13:52.748208 (+ 2990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:52.748216 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:52.748353 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:52.748477 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:52.751476 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:52.752479 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:52.755130 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:52.755743 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:52.755898 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:52.761659 (+ 5761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:52.761668 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:52.761674 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:52.761726 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:52.765596 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:52.766784 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:52.766791 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:52.766795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:52.766956 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:52.767623 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:52.767629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:52.767632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:52.767964 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:52.768110 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:52.769077 (+ 967us) server_negotiation.cc:299] Negotiation successful
0927 15:13:52.769348 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":454,"threads_started":1}
I20260927 15:13:52.770913 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:52.743340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:33393 (local address 127.10.33.65:51913)
0927 15:13:52.744964 (+ 1624us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:52.745033 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:52.747566 (+ 2533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:52.748909 (+ 1343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:52.748926 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:52.750029 (+ 1103us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:52.751170 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:52.751213 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:52.752671 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:52.752676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:52.754885 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:52.754915 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:52.756344 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:52.756354 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:52.756769 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:52.757702 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:52.757749 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:52.761321 (+ 3572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:52.765840 (+ 4519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:52.765852 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:52.765902 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:52.766628 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:52.767120 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:52.767126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:52.767130 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:52.767473 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:52.768194 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:52.768207 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:52.768687 (+ 480us) client_negotiation.cc:769] Sending connection context
0927 15:13:52.768974 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 15:13:52.769596 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"thread_start_us":742,"threads_started":1}
I20260927 15:13:52.784662 18036 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:40973
I20260927 15:13:52.784682 18171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:40973 every 8 connection(s)
I20260927 15:13:52.786384 18036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:13:52.788719 18172 heartbeater.cc:347] Connected to a master server at 127.10.33.126:33393
I20260927 15:13:52.789000 18172 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:52.789793 18172 heartbeater.cc:511] Master 127.10.33.126:33393 requested a full tablet report, sending...
I20260927 15:13:52.790381 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:52.777559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:37545 (local address 127.10.33.126:33393)
0927 15:13:52.777710 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:52.777715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:52.778320 (+ 605us) server_negotiation.cc:407] Connection header received
0927 15:13:52.778481 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:52.778487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:52.778552 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:52.778641 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:52.779554 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:52.780238 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:52.781455 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:52.781879 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:52.782009 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:52.784173 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:52.784178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:52.784182 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:52.784215 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:52.787417 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:52.788423 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:52.788428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:52.788432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:52.788591 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:52.788973 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:52.788978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:52.788982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:52.789270 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:52.789393 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:52.789928 (+ 535us) server_negotiation.cc:299] Negotiation successful
0927 15:13:52.790128 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:13:52.790721 18166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:52.777628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:33393 (local address 127.10.33.65:37545)
0927 15:13:52.778221 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:52.778237 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:52.778373 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:52.778671 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:52.778676 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:52.779140 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:52.779419 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:52.779428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:52.780372 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:52.780376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:52.781329 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:52.781340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:52.782081 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:52.782088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:52.782315 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:52.783063 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:52.783082 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:52.783978 (+ 896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:52.787629 (+ 3651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:52.787638 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:52.787644 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:52.788248 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:52.788707 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:52.788712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:52.788716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:52.788870 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:52.789387 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:52.789392 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:52.789597 (+ 205us) client_negotiation.cc:769] Sending connection context
0927 15:13:52.789784 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 15:13:52.790148 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":340,"threads_started":1}
I20260927 15:13:52.792120 17965 ts_manager.cc:194] Registered new tserver with Master: 82dac9ab7a51481a9bd36d99cac38b25 (127.10.33.65:40973)
I20260927 15:13:52.793721 17965 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:51913
I20260927 15:13:52.796020 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 18036
I20260927 15:13:52.796221 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:13:52.796909 10373 external_mini_cluster.cc:1428] Setting key f692b4a71a53a481373714b8299c30ee
WARNING: no policy specified for oryx/127.10.33.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.33.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.33.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.66.keytab.
Entry for principal oryx/127.10.33.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.66.keytab.
I20260927 15:13:52.842978 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:33393
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.66.keytab
--principal=oryx/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:53.170823 18176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:53.171103 18176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:53.171224 18176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:53.178537 18176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:53.178666 18176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:53.178799 18176 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:13:53.178915 18176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:13:53.189358 18176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.66.keytab
--principal=oryx/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.33.126:33393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:53.190729 18176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:53.192082 18176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:53.206480 18182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:53.206542 18181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:53.206513 18184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:53.207485 18176 server_base.cc:1034] running on GCE node
I20260927 15:13:53.208174 18176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:53.209271 18176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:53.210799 18176 hybrid_clock.cc:648] HybridClock initialized: now 1790522033210681 us; error 65 us; skew 500 ppm
I20260927 15:13:53.211299 18176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:53 dist-test-slave-f5h6 krb5kdc[17920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522033, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:53.215427 18176 init.cc:377] Logged in from keytab as oryx/127.10.33.66@KRBTEST.COM (short username oryx)
I20260927 15:13:53.217319 18176 webserver.cc:460] Webserver started at http://127.10.33.66:43055/ using document root <none> and password file <none>
I20260927 15:13:53.218674 18176 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:53.218967 18176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:53.219389 18176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:53.222636 18176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "5b198eed7bc747ef8b132fa94a6203c8"
format_stamp: "Formatted at 2026-09-27 15:13:53 on dist-test-slave-f5h6"
server_key: "09e7ae965f5461adaedfd7ff5db62256"
server_key_iv: "7de32b89fc78f41fb93702e0f24af90d"
server_key_version: "encryptionkey@0"
I20260927 15:13:53.223506 18176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "5b198eed7bc747ef8b132fa94a6203c8"
format_stamp: "Formatted at 2026-09-27 15:13:53 on dist-test-slave-f5h6"
server_key: "09e7ae965f5461adaedfd7ff5db62256"
server_key_iv: "7de32b89fc78f41fb93702e0f24af90d"
server_key_version: "encryptionkey@0"
I20260927 15:13:53.229315 18176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:13:53.233290 18191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:53.234743 18176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:13:53.234925 18176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "5b198eed7bc747ef8b132fa94a6203c8"
format_stamp: "Formatted at 2026-09-27 15:13:53 on dist-test-slave-f5h6"
server_key: "09e7ae965f5461adaedfd7ff5db62256"
server_key_iv: "7de32b89fc78f41fb93702e0f24af90d"
server_key_version: "encryptionkey@0"
I20260927 15:13:53.235157 18176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:53.263674 18176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:53.268147 18176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:53.268417 18176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:53.271081 18176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:53.271167 18176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:53.271312 18176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:53.271411 18176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 27 15:13:53 dist-test-slave-f5h6 krb5kdc[17920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522033, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM
I20260927 15:13:53.296741 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:53.273064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:39321 (local address 127.10.33.126:33393)
0927 15:13:53.273272 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:53.273278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:53.273566 (+ 288us) server_negotiation.cc:407] Connection header received
0927 15:13:53.276340 (+ 2774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:53.276344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:53.276412 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:53.276528 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:53.279060 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:53.279802 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:53.282749 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:53.283375 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:53.283507 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:53.288881 (+ 5374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:53.288893 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:53.288899 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:53.288947 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:53.292390 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:53.293552 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:53.293558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:53.293562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:53.293711 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:53.294366 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:53.294372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:53.294375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:53.294828 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:53.295016 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:53.296140 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0927 15:13:53.296390 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 15:13:53.297909 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:53.272482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:33393 (local address 127.10.33.66:39321)
0927 15:13:53.273402 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:53.273454 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:53.276050 (+ 2596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:53.276838 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:53.276850 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:53.277731 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:53.278829 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:53.278864 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:53.279968 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:53.279974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:53.282481 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:53.282502 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:53.283649 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:53.283658 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:53.283893 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:53.284936 (+ 1043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:53.284978 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:53.288503 (+ 3525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:53.292599 (+ 4096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:53.292610 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:53.292632 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:53.293367 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:53.293835 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:53.293840 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:53.293845 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:53.294236 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:53.295101 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:53.295109 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:53.295679 (+ 570us) client_negotiation.cc:769] Sending connection context
0927 15:13:53.296003 (+ 324us) client_negotiation.cc:240] Negotiation successful
0927 15:13:53.296605 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":384,"threads_started":1}
I20260927 15:13:53.322739 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:53.305283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:60103 (local address 127.10.33.126:33393)
0927 15:13:53.305485 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:53.305492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:53.306224 (+ 732us) server_negotiation.cc:407] Connection header received
0927 15:13:53.306434 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:53.306440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:53.306521 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:53.306623 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:53.308008 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:53.309034 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:53.311171 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:53.311686 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:53.311816 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:53.315154 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:53.315163 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:53.315168 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:53.315211 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:53.319242 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:53.320436 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:53.320442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:53.320446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:53.320610 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:53.321007 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:53.321010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:53.321013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:53.321448 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:53.321595 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:53.322183 (+ 588us) server_negotiation.cc:299] Negotiation successful
0927 15:13:53.322417 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:13:53.322716 18176 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:44357
I20260927 15:13:53.323010 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:53.305292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:33393 (local address 127.10.33.66:60103)
0927 15:13:53.306104 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:53.306124 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:53.306297 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:53.306699 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:53.306705 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:53.307390 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:53.307801 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:53.307816 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:53.309237 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:53.309241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:53.310995 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:53.311017 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:53.311921 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:53.311928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:53.312103 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:53.312808 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:53.312834 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:53.314793 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:53.319487 (+ 4694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:53.319495 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:53.319500 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:53.320209 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:53.320734 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:53.320739 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:53.320742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:53.320897 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:53.321596 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:53.321602 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:53.321846 (+ 244us) client_negotiation.cc:769] Sending connection context
0927 15:13:53.322050 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 15:13:53.322437 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":443,"threads_started":1}
I20260927 15:13:53.322747 18310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:44357 every 8 connection(s)
I20260927 15:13:53.324287 18176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:13:53.326380 18311 heartbeater.cc:347] Connected to a master server at 127.10.33.126:33393
I20260927 15:13:53.326696 18311 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:53.327355 18311 heartbeater.cc:511] Master 127.10.33.126:33393 requested a full tablet report, sending...
I20260927 15:13:53.328590 17965 ts_manager.cc:194] Registered new tserver with Master: 5b198eed7bc747ef8b132fa94a6203c8 (127.10.33.66:44357)
I20260927 15:13:53.329442 17965 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:39321
I20260927 15:13:53.334211 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 18176
I20260927 15:13:53.334363 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:13:53.334758 10373 external_mini_cluster.cc:1428] Setting key 23cd84bc757e4b8784f5fdd5779c087c
WARNING: no policy specified for oryx/127.10.33.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.33.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.33.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.67.keytab.
Entry for principal oryx/127.10.33.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.67.keytab.
I20260927 15:13:53.382318 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:33393
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.67.keytab
--principal=oryx/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:53.709820 18315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:53.710176 18315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:53.710299 18315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:53.717734 18315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:53.717916 18315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:53.718041 18315 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:13:53.718152 18315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:13:53.728585 18315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.67.keytab
--principal=oryx/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.33.126:33393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:53.729936 18315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:53.731259 18315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:53.744462 18320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:53.744498 18321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:53.744452 18323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:53.745777 18315 server_base.cc:1034] running on GCE node
I20260927 15:13:53.746443 18315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:53.747412 18315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:53.748644 18315 hybrid_clock.cc:648] HybridClock initialized: now 1790522033748571 us; error 45 us; skew 500 ppm
I20260927 15:13:53.748976 18315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:13:53 dist-test-slave-f5h6 krb5kdc[17920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522033, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:13:53.752056 18315 init.cc:377] Logged in from keytab as oryx/127.10.33.67@KRBTEST.COM (short username oryx)
I20260927 15:13:53.753532 18315 webserver.cc:460] Webserver started at http://127.10.33.67:45805/ using document root <none> and password file <none>
I20260927 15:13:53.754264 18315 fs_manager.cc:359] Metadata directory not provided
I20260927 15:13:53.754348 18315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:13:53.754623 18315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:13:53.757519 18315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b"
format_stamp: "Formatted at 2026-09-27 15:13:53 on dist-test-slave-f5h6"
server_key: "ed06a61a4c6a107596b766c7cb575d76"
server_key_iv: "789e6a11499ade4e03bba49bd4ce69c7"
server_key_version: "encryptionkey@0"
I20260927 15:13:53.758328 18315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b"
format_stamp: "Formatted at 2026-09-27 15:13:53 on dist-test-slave-f5h6"
server_key: "ed06a61a4c6a107596b766c7cb575d76"
server_key_iv: "789e6a11499ade4e03bba49bd4ce69c7"
server_key_version: "encryptionkey@0"
I20260927 15:13:53.763648 18315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260927 15:13:53.767294 18330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:53.768667 18315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:13:53.768819 18315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b"
format_stamp: "Formatted at 2026-09-27 15:13:53 on dist-test-slave-f5h6"
server_key: "ed06a61a4c6a107596b766c7cb575d76"
server_key_iv: "789e6a11499ade4e03bba49bd4ce69c7"
server_key_version: "encryptionkey@0"
I20260927 15:13:53.769028 18315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:13:53.796950 18172 heartbeater.cc:503] Master 127.10.33.126:33393 was elected leader, sending a full tablet report...
I20260927 15:13:53.797456 18315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:13:53.802491 18315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:13:53.802719 18315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:13:53.805341 18315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:13:53.805436 18315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:13:53.805567 18315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:13:53.805624 18315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:13:53 dist-test-slave-f5h6 krb5kdc[17920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522033, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM
I20260927 15:13:53.831192 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:53.805979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:52659 (local address 127.10.33.126:33393)
0927 15:13:53.806160 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:53.806166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:53.807659 (+ 1493us) server_negotiation.cc:407] Connection header received
0927 15:13:53.810460 (+ 2801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:53.810467 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:53.810562 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:53.810640 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:53.813691 (+ 3051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:53.814757 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:53.817682 (+ 2925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:53.818203 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:53.818324 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:53.823578 (+ 5254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:53.823586 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:53.823589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:53.823634 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:53.826474 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:53.827757 (+ 1283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:53.827763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:53.827766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:53.828029 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:53.828678 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:53.828684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:53.828688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:53.829284 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:53.829445 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:53.830576 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0927 15:13:53.830843 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:13:53.832485 18339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:53.806588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:33393 (local address 127.10.33.67:52659)
0927 15:13:53.807459 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:53.807511 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:53.810044 (+ 2533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:53.811269 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:53.811286 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:53.812269 (+ 983us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:53.813412 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:53.813451 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:53.815038 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:53.815044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:53.817439 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:53.817463 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:53.818509 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:53.818519 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:53.818851 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:53.819728 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:53.819753 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:53.823141 (+ 3388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:53.826743 (+ 3602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:53.826754 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:53.826782 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:53.827598 (+ 816us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:53.828207 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:53.828213 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:53.828220 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:53.828555 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:53.829531 (+ 976us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:53.829540 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:53.830083 (+ 543us) client_negotiation.cc:769] Sending connection context
0927 15:13:53.830410 (+ 327us) client_negotiation.cc:240] Negotiation successful
0927 15:13:53.831011 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":342,"threads_started":1}
I20260927 15:13:53.854517 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:53.838443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:60275 (local address 127.10.33.126:33393)
0927 15:13:53.838610 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:53.838614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:53.839304 (+ 690us) server_negotiation.cc:407] Connection header received
0927 15:13:53.839492 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:53.839499 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:53.839570 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:53.839671 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:53.840778 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:53.841498 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:53.843287 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:53.843882 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:53.844027 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:53.847836 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:53.847844 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:53.847848 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:53.847908 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:53.851300 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:53.852207 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:53.852213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:53.852224 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:53.852402 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:53.852912 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:53.852917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:53.852921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:53.853198 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:53.853330 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:53.853959 (+ 629us) server_negotiation.cc:299] Negotiation successful
0927 15:13:53.854167 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:13:53.854712 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:53.838476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:33393 (local address 127.10.33.67:60275)
0927 15:13:53.839188 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:53.839206 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:53.839367 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:53.839708 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:53.839712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:53.840279 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:53.840608 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:53.840620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:53.841648 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:53.841654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:53.843085 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:53.843100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:53.844810 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:53.844818 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:53.845097 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:53.846363 (+ 1266us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:53.846386 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:53.847531 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:53.851500 (+ 3969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:53.851506 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:53.851510 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:53.852078 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:53.852562 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:53.852567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:53.852571 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:53.852764 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:53.853354 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:53.853360 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:53.853604 (+ 244us) client_negotiation.cc:769] Sending connection context
0927 15:13:53.853770 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 15:13:53.854211 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":397,"threads_started":1}
I20260927 15:13:53.863384 18315 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:36271
I20260927 15:13:53.863441 18449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:36271 every 8 connection(s)
I20260927 15:13:53.864591 18315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:13:53.864898 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 18315
I20260927 15:13:53.865087 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:13:53.865669 10373 external_mini_cluster.cc:1428] Setting key c72c8c3066403a5fbc9d4cede17d775c
I20260927 15:13:53.866878 18450 heartbeater.cc:347] Connected to a master server at 127.10.33.126:33393
I20260927 15:13:53.867125 18450 heartbeater.cc:464] Registering TS with master...
I20260927 15:13:53.867841 18450 heartbeater.cc:511] Master 127.10.33.126:33393 requested a full tablet report, sending...
I20260927 15:13:53.869201 17964 ts_manager.cc:194] Registered new tserver with Master: 7ac5cf19eb0a4149b1aa89784b1d9f8b (127.10.33.67:36271)
I20260927 15:13:53.870143 17964 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:52659
I20260927 15:13:53.881161 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:13:53 dist-test-slave-f5h6 krb5kdc[17920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522033, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:13:53 dist-test-slave-f5h6 krb5kdc[17920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:13:53 dist-test-slave-f5h6 krb5kdc[17920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:13:54.332175 18311 heartbeater.cc:503] Master 127.10.33.126:33393 was elected leader, sending a full tablet report...
I20260927 15:13:54.872669 18450 heartbeater.cc:503] Master 127.10.33.126:33393 was elected leader, sending a full tablet report...
W20260927 15:13:54.900761 18458 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.975s user 0.003s sys 0.011s
W20260927 15:13:54.902010 18060 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:13:53.920112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37630 (local address 127.10.33.126:33393)
0927 15:13:53.920286 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:53.920291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:53.920838 (+ 547us) server_negotiation.cc:407] Connection header received
0927 15:13:53.921015 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:53.921021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:53.921082 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:53.921165 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:53.922531 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:53.923242 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:53.924452 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:53.924876 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:53.925000 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:54.901654 (+976654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37630: BlockingRecv error: recv got EOF from 127.0.0.1:37630 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 15:13:54 dist-test-slave-f5h6 krb5kdc[17920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522033, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM
I20260927 15:13:54.920528 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:54.905750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37644 (local address 127.10.33.126:33393)
0927 15:13:54.905984 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:54.905988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:54.906432 (+ 444us) server_negotiation.cc:407] Connection header received
0927 15:13:54.906602 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:54.906608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:54.906700 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:54.906787 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:54.908110 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:54.908847 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:54.910345 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:54.910806 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:54.910952 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:54.915210 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:54.915219 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:54.915225 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:54.915272 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:54.917941 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:54.918682 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:54.918686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:54.918689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:54.918825 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:54.919172 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:54.919175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:54.919177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:54.919449 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:54.919741 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:54.919991 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 15:13:54.920212 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:13:54.924403 17964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:13:54.931459 17964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:13:54.967271 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:54.944667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:44357 (local address 127.0.0.1:45924)
0927 15:13:54.958486 (+ 13819us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:54.958509 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:54.958756 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:54.959713 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:54.959717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:54.959747 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:54.960635 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:54.960657 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:54.962729 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:54.962734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:54.964656 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:54.964666 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:54.966212 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:54.966218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:54.966483 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:54.966528 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 15:13:54.966635 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 15:13:54.966825 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13697,"mutex_wait_us":98,"thread_start_us":13328,"threads_started":1}
I20260927 15:13:54.967859 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:54.944610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:40973 (local address 127.0.0.1:40252)
0927 15:13:54.957454 (+ 12844us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:54.957508 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:54.957742 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:54.958762 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:54.958770 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:54.958796 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:54.960845 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:54.960858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:54.962689 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:54.962693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:54.965459 (+ 2766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:54.965473 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:54.967154 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:54.967161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:54.967319 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:54.967348 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 15:13:54.967434 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 15:13:54.967552 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12547,"mutex_wait_us":151,"thread_start_us":13572,"threads_started":1}
I20260927 15:13:54.968173 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:54.945078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45924 (local address 127.10.33.66:44357)
0927 15:13:54.957351 (+ 12273us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:54.957360 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:54.958625 (+ 1265us) server_negotiation.cc:407] Connection header received
0927 15:13:54.958954 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:54.958962 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:54.959212 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:54.959404 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:54.960916 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:54.962529 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:54.964806 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:54.966012 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:54.966131 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:54.966349 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:54.967694 (+ 1345us) server_negotiation.cc:299] Negotiation successful
0927 15:13:54.967899 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12067,"thread_start_us":12290,"threads_started":1}
I20260927 15:13:54.968530 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:54.944841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40252 (local address 127.10.33.65:40973)
0927 15:13:54.958008 (+ 13167us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:54.958016 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:54.958046 (+ 30us) server_negotiation.cc:407] Connection header received
0927 15:13:54.958147 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:54.958154 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:54.958434 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:54.958626 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:54.961084 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:54.962565 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:54.965641 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:54.966951 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:54.967080 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:54.967259 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:54.968090 (+ 831us) server_negotiation.cc:299] Negotiation successful
0927 15:13:54.968262 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12995,"thread_start_us":12863,"threads_started":1}
I20260927 15:13:54.969520 18473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:54.959150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.10.33.67:36271)
0927 15:13:54.959947 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:54.959958 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:54.961103 (+ 1145us) server_negotiation.cc:407] Connection header received
0927 15:13:54.961519 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:54.961527 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:54.961902 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:54.962086 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:54.962679 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:54.964193 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:54.966686 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:54.968073 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:54.968202 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:54.968387 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:54.969133 (+ 746us) server_negotiation.cc:299] Negotiation successful
0927 15:13:54.969291 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":377,"threads_started":1}
I20260927 15:13:54.969519 18474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:54.959904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:36271 (local address 127.0.0.1:33180)
0927 15:13:54.960942 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:54.960969 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:54.961285 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:54.962158 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:54.962163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:54.962188 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:54.962519 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:54.962528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:54.964363 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:54.964369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:54.966523 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:54.966538 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:54.968789 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:54.968795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:54.968950 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:54.968982 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 15:13:54.969070 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 15:13:54.969192 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":52,"thread_start_us":662,"threads_started":1}
I20260927 15:13:54.971849 18102 tablet_service.cc:1467] Processing CreateTablet for tablet 9c35c234a28b4c168dfe7ee403566324 (DEFAULT_TABLE table=test-table [id=4b30e3010741408587568b54e5803492]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:54.971850 18241 tablet_service.cc:1467] Processing CreateTablet for tablet 9c35c234a28b4c168dfe7ee403566324 (DEFAULT_TABLE table=test-table [id=4b30e3010741408587568b54e5803492]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:54.973032 18241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c35c234a28b4c168dfe7ee403566324. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:54.973040 18102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c35c234a28b4c168dfe7ee403566324. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:54.976881 18380 tablet_service.cc:1467] Processing CreateTablet for tablet 9c35c234a28b4c168dfe7ee403566324 (DEFAULT_TABLE table=test-table [id=4b30e3010741408587568b54e5803492]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:54.978063 18380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c35c234a28b4c168dfe7ee403566324. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:54.984038 18475 tablet_bootstrap.cc:492] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8: Bootstrap starting.
I20260927 15:13:54.987473 18475 tablet_bootstrap.cc:654] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:54.989297 18475 log.cc:824] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:54.996016 18478 tablet_bootstrap.cc:492] T 9c35c234a28b4c168dfe7ee403566324 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Bootstrap starting.
I20260927 15:13:55.003051 18477 tablet_bootstrap.cc:492] T 9c35c234a28b4c168dfe7ee403566324 P 82dac9ab7a51481a9bd36d99cac38b25: Bootstrap starting.
I20260927 15:13:55.004339 18475 tablet_bootstrap.cc:492] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8: No bootstrap required, opened a new log
I20260927 15:13:55.004726 18475 ts_tablet_manager.cc:1397] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.004s
I20260927 15:13:55.006551 18478 tablet_bootstrap.cc:654] T 9c35c234a28b4c168dfe7ee403566324 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:55.007947 18477 tablet_bootstrap.cc:654] T 9c35c234a28b4c168dfe7ee403566324 P 82dac9ab7a51481a9bd36d99cac38b25: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:55.008051 18478 log.cc:824] T 9c35c234a28b4c168dfe7ee403566324 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:55.009239 18477 log.cc:824] T 9c35c234a28b4c168dfe7ee403566324 P 82dac9ab7a51481a9bd36d99cac38b25: Log is configured to *not* fsync() on all Append() calls
I20260927 15:13:55.011708 18478 tablet_bootstrap.cc:492] T 9c35c234a28b4c168dfe7ee403566324 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: No bootstrap required, opened a new log
I20260927 15:13:55.012074 18478 ts_tablet_manager.cc:1397] T 9c35c234a28b4c168dfe7ee403566324 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20260927 15:13:55.014678 18475 raft_consensus.cc:348] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.015066 18475 raft_consensus.cc:374] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:55.015174 18475 raft_consensus.cc:729] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b198eed7bc747ef8b132fa94a6203c8, State: Initialized, Role: FOLLOWER
I20260927 15:13:55.015672 18477 tablet_bootstrap.cc:492] T 9c35c234a28b4c168dfe7ee403566324 P 82dac9ab7a51481a9bd36d99cac38b25: No bootstrap required, opened a new log
I20260927 15:13:55.015746 18475 consensus_queue.cc:260] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.015945 18477 ts_tablet_manager.cc:1397] T 9c35c234a28b4c168dfe7ee403566324 P 82dac9ab7a51481a9bd36d99cac38b25: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260927 15:13:55.017468 18475 ts_tablet_manager.cc:1428] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260927 15:13:55.022641 18478 raft_consensus.cc:348] T 9c35c234a28b4c168dfe7ee403566324 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.023128 18478 raft_consensus.cc:374] T 9c35c234a28b4c168dfe7ee403566324 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:55.023223 18478 raft_consensus.cc:729] T 9c35c234a28b4c168dfe7ee403566324 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac5cf19eb0a4149b1aa89784b1d9f8b, State: Initialized, Role: FOLLOWER
I20260927 15:13:55.023834 18478 consensus_queue.cc:260] T 9c35c234a28b4c168dfe7ee403566324 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.025676 18478 ts_tablet_manager.cc:1428] T 9c35c234a28b4c168dfe7ee403566324 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260927 15:13:55.025974 18477 raft_consensus.cc:348] T 9c35c234a28b4c168dfe7ee403566324 P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.026418 18477 raft_consensus.cc:374] T 9c35c234a28b4c168dfe7ee403566324 P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:55.026530 18477 raft_consensus.cc:729] T 9c35c234a28b4c168dfe7ee403566324 P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82dac9ab7a51481a9bd36d99cac38b25, State: Initialized, Role: FOLLOWER
I20260927 15:13:55.027156 18477 consensus_queue.cc:260] T 9c35c234a28b4c168dfe7ee403566324 P 82dac9ab7a51481a9bd36d99cac38b25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.030741 18477 ts_tablet_manager.cc:1428] T 9c35c234a28b4c168dfe7ee403566324 P 82dac9ab7a51481a9bd36d99cac38b25: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
W20260927 15:13:55.040650 18173 tablet.cc:2369] T 9c35c234a28b4c168dfe7ee403566324 P 82dac9ab7a51481a9bd36d99cac38b25: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:13:55.077768 18312 tablet.cc:2369] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:13:55.080744 18481 raft_consensus.cc:482] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:13:55.081002 18481 raft_consensus.cc:504] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.082677 18481 leader_election.cc:288] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82dac9ab7a51481a9bd36d99cac38b25 (127.10.33.65:40973), 7ac5cf19eb0a4149b1aa89784b1d9f8b (127.10.33.67:36271)
I20260927 15:13:55.089735 18473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.083127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:48071 (local address 127.10.33.67:36271)
0927 15:13:55.083348 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:55.083355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:55.084131 (+ 776us) server_negotiation.cc:407] Connection header received
0927 15:13:55.084314 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:55.084318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:55.084395 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:55.084470 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:55.085035 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.086015 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.088053 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.088818 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.088967 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.089044 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:55.089363 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 15:13:55.089497 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:13:55.089752 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.083330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:58545 (local address 127.10.33.65:40973)
0927 15:13:55.083518 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:55.083524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:55.084235 (+ 711us) server_negotiation.cc:407] Connection header received
0927 15:13:55.084392 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:55.084396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:55.084505 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:55.084583 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:55.085200 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.086065 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.088104 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.088913 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.089060 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.089132 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:55.089403 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 15:13:55.089527 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:13:55.090124 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.082951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:36271 (local address 127.10.33.66:48071)
0927 15:13:55.084009 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:55.084025 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:55.084186 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:55.084499 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:55.084504 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:55.084526 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:55.084880 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:55.084896 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.086142 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.086147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:55.087901 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:55.087916 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.089025 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.089031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:55.089183 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.089207 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:13:55.089294 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 15:13:55.089430 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":39,"thread_start_us":594,"threads_started":1}
I20260927 15:13:55.090176 18485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.082989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:40973 (local address 127.10.33.66:58545)
0927 15:13:55.084132 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:55.084146 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:55.084273 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:55.084620 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:55.084625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:55.084691 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:55.085045 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:55.085054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.086242 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.086249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:55.087953 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:55.087965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.089124 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.089133 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:55.089259 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.089290 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 15:13:55.089366 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 15:13:55.089498 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"spinlock_wait_cycles":35200,"thread_start_us":836,"threads_started":1}
I20260927 15:13:55.091208 18122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c35c234a28b4c168dfe7ee403566324" candidate_uuid: "5b198eed7bc747ef8b132fa94a6203c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82dac9ab7a51481a9bd36d99cac38b25" is_pre_election: true
I20260927 15:13:55.091208 18404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c35c234a28b4c168dfe7ee403566324" candidate_uuid: "5b198eed7bc747ef8b132fa94a6203c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" is_pre_election: true
I20260927 15:13:55.091787 18404 raft_consensus.cc:2454] T 9c35c234a28b4c168dfe7ee403566324 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b198eed7bc747ef8b132fa94a6203c8 in term 0.
I20260927 15:13:55.091799 18122 raft_consensus.cc:2454] T 9c35c234a28b4c168dfe7ee403566324 P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b198eed7bc747ef8b132fa94a6203c8 in term 0.
I20260927 15:13:55.092470 18193 leader_election.cc:302] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b198eed7bc747ef8b132fa94a6203c8, 82dac9ab7a51481a9bd36d99cac38b25; no voters:
I20260927 15:13:55.092945 18481 raft_consensus.cc:2781] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:13:55.093070 18481 raft_consensus.cc:482] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:13:55.093168 18481 raft_consensus.cc:3037] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:55.095378 18481 raft_consensus.cc:504] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.095975 18481 leader_election.cc:288] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8 [CANDIDATE]: Term 1 election: Requested vote from peers 82dac9ab7a51481a9bd36d99cac38b25 (127.10.33.65:40973), 7ac5cf19eb0a4149b1aa89784b1d9f8b (127.10.33.67:36271)
I20260927 15:13:55.096593 18404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c35c234a28b4c168dfe7ee403566324" candidate_uuid: "5b198eed7bc747ef8b132fa94a6203c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b"
I20260927 15:13:55.096593 18122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c35c234a28b4c168dfe7ee403566324" candidate_uuid: "5b198eed7bc747ef8b132fa94a6203c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82dac9ab7a51481a9bd36d99cac38b25"
I20260927 15:13:55.096812 18122 raft_consensus.cc:3037] T 9c35c234a28b4c168dfe7ee403566324 P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:55.096810 18404 raft_consensus.cc:3037] T 9c35c234a28b4c168dfe7ee403566324 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:55.099967 18404 raft_consensus.cc:2454] T 9c35c234a28b4c168dfe7ee403566324 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b198eed7bc747ef8b132fa94a6203c8 in term 1.
I20260927 15:13:55.099967 18122 raft_consensus.cc:2454] T 9c35c234a28b4c168dfe7ee403566324 P 82dac9ab7a51481a9bd36d99cac38b25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b198eed7bc747ef8b132fa94a6203c8 in term 1.
I20260927 15:13:55.100570 18195 leader_election.cc:302] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5b198eed7bc747ef8b132fa94a6203c8, 7ac5cf19eb0a4149b1aa89784b1d9f8b, 82dac9ab7a51481a9bd36d99cac38b25; no voters:
I20260927 15:13:55.100946 18481 raft_consensus.cc:2781] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:55.101450 18481 raft_consensus.cc:686] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8 [term 1 LEADER]: Becoming Leader. State: Replica: 5b198eed7bc747ef8b132fa94a6203c8, State: Running, Role: LEADER
I20260927 15:13:55.102104 18481 consensus_queue.cc:237] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.107007 17964 catalog_manager.cc:5495] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b198eed7bc747ef8b132fa94a6203c8 (127.10.33.66). New cstate: current_term: 1 leader_uuid: "5b198eed7bc747ef8b132fa94a6203c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } health_report { overall_health: UNKNOWN } } }
W20260927 15:13:55.116989 18451 tablet.cc:2369] T 9c35c234a28b4c168dfe7ee403566324 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:13:55.128536 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.123465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45926 (local address 127.10.33.66:44357)
0927 15:13:55.123654 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:55.123659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:55.123679 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:13:55.123850 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:55.123854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:55.123948 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:55.124076 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:13:55.124440 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.125150 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.126509 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.126960 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.127063 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.127463 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:13:55.127988 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:13:55.128109 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:55.128207 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 15:13:55.128288 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:13:55.136758 18404 raft_consensus.cc:1261] T 9c35c234a28b4c168dfe7ee403566324 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 1 FOLLOWER]: Refusing update from remote peer 5b198eed7bc747ef8b132fa94a6203c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:55.136835 18122 raft_consensus.cc:1261] T 9c35c234a28b4c168dfe7ee403566324 P 82dac9ab7a51481a9bd36d99cac38b25 [term 1 FOLLOWER]: Refusing update from remote peer 5b198eed7bc747ef8b132fa94a6203c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:55.137996 18481 consensus_queue.cc:1035] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:55.138285 18486 consensus_queue.cc:1035] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:55.167122 17965 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:55.167385 17965 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:55.171000 17965 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a: Sending DeleteTablet for 3 replicas of tablet 9c35c234a28b4c168dfe7ee403566324
I20260927 15:13:55.172310 18102 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c35c234a28b4c168dfe7ee403566324 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:55 UTC) from {username='oryx', principal='oryx/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:40252
I20260927 15:13:55.173398 18380 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c35c234a28b4c168dfe7ee403566324 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:55 UTC) from {username='oryx', principal='oryx/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:33180
I20260927 15:13:55.174243 18499 tablet_replica.cc:330] T 9c35c234a28b4c168dfe7ee403566324 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: stopping tablet replica
I20260927 15:13:55.174371 18241 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c35c234a28b4c168dfe7ee403566324 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:55 UTC) from {username='oryx', principal='oryx/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:45924
I20260927 15:13:55.174827 18499 raft_consensus.cc:2229] T 9c35c234a28b4c168dfe7ee403566324 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:55.175384 18499 raft_consensus.cc:2258] T 9c35c234a28b4c168dfe7ee403566324 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:55.177588 18499 ts_tablet_manager.cc:1905] T 9c35c234a28b4c168dfe7ee403566324 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:55.179626 17965 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:13:55.180349 17965 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:13:55.185181 18499 ts_tablet_manager.cc:1918] T 9c35c234a28b4c168dfe7ee403566324 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:13:55.185552 18499 log.cc:1196] T 9c35c234a28b4c168dfe7ee403566324 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/9c35c234a28b4c168dfe7ee403566324
I20260927 15:13:55.186172 18499 ts_tablet_manager.cc:1939] T 9c35c234a28b4c168dfe7ee403566324 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Deleting consensus metadata
I20260927 15:13:55.188369 17952 catalog_manager.cc:4841] TS 7ac5cf19eb0a4149b1aa89784b1d9f8b (127.10.33.67:36271): tablet 9c35c234a28b4c168dfe7ee403566324 (table test-table [id=4b30e3010741408587568b54e5803492]) successfully deleted
I20260927 15:13:55.194214 18101 tablet_service.cc:1467] Processing CreateTablet for tablet a2c232ae01d943f3b59ae37e908404be (DEFAULT_TABLE table=test-table [id=97f668f418624fb78dbc140bdc71b8d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:55.194859 18101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2c232ae01d943f3b59ae37e908404be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:55.195657 18240 tablet_service.cc:1467] Processing CreateTablet for tablet a2c232ae01d943f3b59ae37e908404be (DEFAULT_TABLE table=test-table [id=97f668f418624fb78dbc140bdc71b8d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:55.196240 18240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2c232ae01d943f3b59ae37e908404be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:55.200229 18380 tablet_service.cc:1467] Processing CreateTablet for tablet a2c232ae01d943f3b59ae37e908404be (DEFAULT_TABLE table=test-table [id=97f668f418624fb78dbc140bdc71b8d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:55.200691 18380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2c232ae01d943f3b59ae37e908404be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:55.201262 18477 tablet_bootstrap.cc:492] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25: Bootstrap starting.
I20260927 15:13:55.203047 18477 tablet_bootstrap.cc:654] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:55.206036 18500 tablet_replica.cc:330] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8: stopping tablet replica
I20260927 15:13:55.206526 18500 raft_consensus.cc:2229] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:13:55.207198 18477 tablet_bootstrap.cc:492] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25: No bootstrap required, opened a new log
I20260927 15:13:55.207207 18500 raft_consensus.cc:2258] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:55.207365 18477 ts_tablet_manager.cc:1397] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 15:13:55.207422 18475 tablet_bootstrap.cc:492] T a2c232ae01d943f3b59ae37e908404be P 5b198eed7bc747ef8b132fa94a6203c8: Bootstrap starting.
I20260927 15:13:55.208012 18498 tablet_replica.cc:330] T 9c35c234a28b4c168dfe7ee403566324 P 82dac9ab7a51481a9bd36d99cac38b25: stopping tablet replica
I20260927 15:13:55.208170 18477 raft_consensus.cc:348] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.208388 18477 raft_consensus.cc:374] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:55.208487 18477 raft_consensus.cc:729] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82dac9ab7a51481a9bd36d99cac38b25, State: Initialized, Role: FOLLOWER
I20260927 15:13:55.208772 18477 consensus_queue.cc:260] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.208988 18475 tablet_bootstrap.cc:654] T a2c232ae01d943f3b59ae37e908404be P 5b198eed7bc747ef8b132fa94a6203c8: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:55.209414 18477 ts_tablet_manager.cc:1428] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:13:55.210294 18500 ts_tablet_manager.cc:1905] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:55.208391 18498 raft_consensus.cc:2229] T 9c35c234a28b4c168dfe7ee403566324 P 82dac9ab7a51481a9bd36d99cac38b25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:55.212003 18498 raft_consensus.cc:2258] T 9c35c234a28b4c168dfe7ee403566324 P 82dac9ab7a51481a9bd36d99cac38b25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:55.215314 18498 ts_tablet_manager.cc:1905] T 9c35c234a28b4c168dfe7ee403566324 P 82dac9ab7a51481a9bd36d99cac38b25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:55.217298 18478 tablet_bootstrap.cc:492] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Bootstrap starting.
I20260927 15:13:55.219031 18478 tablet_bootstrap.cc:654] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:55.221306 18478 tablet_bootstrap.cc:492] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b: No bootstrap required, opened a new log
I20260927 15:13:55.221477 18478 ts_tablet_manager.cc:1397] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 15:13:55.222290 18478 raft_consensus.cc:348] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.222498 18478 raft_consensus.cc:374] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:55.222577 18478 raft_consensus.cc:729] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac5cf19eb0a4149b1aa89784b1d9f8b, State: Initialized, Role: FOLLOWER
I20260927 15:13:55.222872 18478 consensus_queue.cc:260] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.223528 18478 ts_tablet_manager.cc:1428] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:13:55.224028 18500 ts_tablet_manager.cc:1918] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:13:55.224220 18500 log.cc:1196] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/9c35c234a28b4c168dfe7ee403566324
I20260927 15:13:55.224339 18498 ts_tablet_manager.cc:1918] T 9c35c234a28b4c168dfe7ee403566324 P 82dac9ab7a51481a9bd36d99cac38b25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:13:55.224501 18498 log.cc:1196] T 9c35c234a28b4c168dfe7ee403566324 P 82dac9ab7a51481a9bd36d99cac38b25: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/9c35c234a28b4c168dfe7ee403566324
I20260927 15:13:55.224749 18500 ts_tablet_manager.cc:1939] T 9c35c234a28b4c168dfe7ee403566324 P 5b198eed7bc747ef8b132fa94a6203c8: Deleting consensus metadata
I20260927 15:13:55.224997 18498 ts_tablet_manager.cc:1939] T 9c35c234a28b4c168dfe7ee403566324 P 82dac9ab7a51481a9bd36d99cac38b25: Deleting consensus metadata
I20260927 15:13:55.226567 17952 catalog_manager.cc:4841] TS 5b198eed7bc747ef8b132fa94a6203c8 (127.10.33.66:44357): tablet 9c35c234a28b4c168dfe7ee403566324 (table test-table [id=4b30e3010741408587568b54e5803492]) successfully deleted
I20260927 15:13:55.226567 17950 catalog_manager.cc:4841] TS 82dac9ab7a51481a9bd36d99cac38b25 (127.10.33.65:40973): tablet 9c35c234a28b4c168dfe7ee403566324 (table test-table [id=4b30e3010741408587568b54e5803492]) successfully deleted
I20260927 15:13:55.227916 18475 tablet_bootstrap.cc:492] T a2c232ae01d943f3b59ae37e908404be P 5b198eed7bc747ef8b132fa94a6203c8: No bootstrap required, opened a new log
I20260927 15:13:55.228080 18475 ts_tablet_manager.cc:1397] T a2c232ae01d943f3b59ae37e908404be P 5b198eed7bc747ef8b132fa94a6203c8: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.014s
I20260927 15:13:55.228749 18475 raft_consensus.cc:348] T a2c232ae01d943f3b59ae37e908404be P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.228933 18475 raft_consensus.cc:374] T a2c232ae01d943f3b59ae37e908404be P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:55.229028 18475 raft_consensus.cc:729] T a2c232ae01d943f3b59ae37e908404be P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b198eed7bc747ef8b132fa94a6203c8, State: Initialized, Role: FOLLOWER
I20260927 15:13:55.229305 18475 consensus_queue.cc:260] T a2c232ae01d943f3b59ae37e908404be P 5b198eed7bc747ef8b132fa94a6203c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.229831 18475 ts_tablet_manager.cc:1428] T a2c232ae01d943f3b59ae37e908404be P 5b198eed7bc747ef8b132fa94a6203c8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 15:13:55.275388 18482 raft_consensus.cc:482] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:13:55.275836 18482 raft_consensus.cc:504] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.278119 18482 leader_election.cc:288] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82dac9ab7a51481a9bd36d99cac38b25 (127.10.33.65:40973), 5b198eed7bc747ef8b132fa94a6203c8 (127.10.33.66:44357)
I20260927 15:13:55.286581 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.278346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:44357 (local address 127.10.33.67:58119)
0927 15:13:55.279421 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:55.279442 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:55.279658 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:55.280209 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:55.280214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:55.280257 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:55.280653 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:55.280668 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.282393 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.282398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:55.284448 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:55.284461 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.285723 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.285729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:55.285950 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.285976 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:13:55.286065 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 15:13:55.286209 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"thread_start_us":3295,"threads_started":1}
I20260927 15:13:55.286586 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.278699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:58119 (local address 127.10.33.66:44357)
0927 15:13:55.278896 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:55.278903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:55.279536 (+ 633us) server_negotiation.cc:407] Connection header received
0927 15:13:55.279826 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:55.279832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:55.279909 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:55.279987 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:55.280869 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.282234 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.284614 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.285491 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.285658 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.285737 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:55.286116 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 15:13:55.286263 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 15:13:55.287678 18261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c232ae01d943f3b59ae37e908404be" candidate_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b198eed7bc747ef8b132fa94a6203c8" is_pre_election: true
I20260927 15:13:55.288162 18261 raft_consensus.cc:2454] T a2c232ae01d943f3b59ae37e908404be P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac5cf19eb0a4149b1aa89784b1d9f8b in term 0.
I20260927 15:13:55.288515 18483 raft_consensus.cc:482] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:13:55.288776 18483 raft_consensus.cc:504] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.288938 18334 leader_election.cc:302] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b198eed7bc747ef8b132fa94a6203c8, 7ac5cf19eb0a4149b1aa89784b1d9f8b; no voters:
I20260927 15:13:55.289258 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.278967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:49079 (local address 127.10.33.65:40973)
0927 15:13:55.279149 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:55.279155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:55.282079 (+ 2924us) server_negotiation.cc:407] Connection header received
0927 15:13:55.282231 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:55.282235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:55.282294 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:55.282369 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:55.283114 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.284149 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.286601 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.287882 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.288066 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.288141 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:55.288664 (+ 523us) server_negotiation.cc:299] Negotiation successful
0927 15:13:55.288881 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:13:55.289350 18482 raft_consensus.cc:2781] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:13:55.289451 18482 raft_consensus.cc:482] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:13:55.289541 18482 raft_consensus.cc:3037] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:55.290457 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.278479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:40973 (local address 127.10.33.67:49079)
0927 15:13:55.281970 (+ 3491us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:55.281988 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:55.282108 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:55.282451 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:55.282456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:55.282487 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:55.282935 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:55.282951 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.284326 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.284333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:55.286416 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:55.286431 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.288191 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.288198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:55.288466 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.288501 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 15:13:55.289669 (+ 1168us) client_negotiation.cc:240] Negotiation successful
0927 15:13:55.289829 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3347,"thread_start_us":1899,"threads_started":1}
I20260927 15:13:55.290748 18483 leader_election.cc:288] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b198eed7bc747ef8b132fa94a6203c8 (127.10.33.66:44357), 7ac5cf19eb0a4149b1aa89784b1d9f8b (127.10.33.67:36271)
I20260927 15:13:55.291666 18122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c232ae01d943f3b59ae37e908404be" candidate_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82dac9ab7a51481a9bd36d99cac38b25" is_pre_election: true
I20260927 15:13:55.291920 18122 raft_consensus.cc:2454] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac5cf19eb0a4149b1aa89784b1d9f8b in term 0.
I20260927 15:13:55.292873 18482 raft_consensus.cc:504] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.293987 18122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c232ae01d943f3b59ae37e908404be" candidate_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82dac9ab7a51481a9bd36d99cac38b25"
I20260927 15:13:55.294204 18122 raft_consensus.cc:3037] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:55.297844 18122 raft_consensus.cc:2454] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac5cf19eb0a4149b1aa89784b1d9f8b in term 1.
I20260927 15:13:55.298806 18332 leader_election.cc:302] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ac5cf19eb0a4149b1aa89784b1d9f8b, 82dac9ab7a51481a9bd36d99cac38b25; no voters:
I20260927 15:13:55.300436 18261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c232ae01d943f3b59ae37e908404be" candidate_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b198eed7bc747ef8b132fa94a6203c8"
I20260927 15:13:55.300683 18261 raft_consensus.cc:3037] T a2c232ae01d943f3b59ae37e908404be P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:55.302057 18508 raft_consensus.cc:2781] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:55.303689 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.291979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:46237 (local address 127.10.33.66:44357)
0927 15:13:55.292827 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:55.292834 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:55.293410 (+ 576us) server_negotiation.cc:407] Connection header received
0927 15:13:55.293692 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:55.293697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:55.293769 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:55.293830 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:55.294704 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.295814 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.299300 (+ 3486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.300709 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.300865 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.300944 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:55.303216 (+ 2272us) server_negotiation.cc:299] Negotiation successful
0927 15:13:55.303382 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720}
I20260927 15:13:55.304114 18261 raft_consensus.cc:2454] T a2c232ae01d943f3b59ae37e908404be P 5b198eed7bc747ef8b132fa94a6203c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac5cf19eb0a4149b1aa89784b1d9f8b in term 1.
I20260927 15:13:55.304664 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.291272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:44357 (local address 127.10.33.65:46237)
0927 15:13:55.293293 (+ 2021us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:55.293314 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:55.293515 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:55.294058 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:55.294062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:55.294085 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:55.294523 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:55.294536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.295994 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.295999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:55.299087 (+ 3088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:55.299108 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.302754 (+ 3646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.302761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:55.303001 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.303039 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 15:13:55.304083 (+ 1044us) client_negotiation.cc:240] Negotiation successful
0927 15:13:55.304305 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1897,"thread_start_us":1978,"threads_started":1}
I20260927 15:13:55.304699 18482 leader_election.cc:288] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b [CANDIDATE]: Term 1 election: Requested vote from peers 82dac9ab7a51481a9bd36d99cac38b25 (127.10.33.65:40973), 5b198eed7bc747ef8b132fa94a6203c8 (127.10.33.66:44357)
I20260927 15:13:55.305204 18508 raft_consensus.cc:686] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 1 LEADER]: Becoming Leader. State: Replica: 7ac5cf19eb0a4149b1aa89784b1d9f8b, State: Running, Role: LEADER
I20260927 15:13:55.305709 18508 consensus_queue.cc:237] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.307560 18261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c232ae01d943f3b59ae37e908404be" candidate_uuid: "82dac9ab7a51481a9bd36d99cac38b25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b198eed7bc747ef8b132fa94a6203c8" is_pre_election: true
I20260927 15:13:55.307847 18261 raft_consensus.cc:2379] T a2c232ae01d943f3b59ae37e908404be P 5b198eed7bc747ef8b132fa94a6203c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82dac9ab7a51481a9bd36d99cac38b25 in current term 1: Already voted for candidate 7ac5cf19eb0a4149b1aa89784b1d9f8b in this term.
I20260927 15:13:55.312182 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.293652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:36271 (local address 127.10.33.65:38865)
0927 15:13:55.305136 (+ 11484us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:55.305156 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:55.305313 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:55.305844 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:55.305850 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:55.305914 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:13:55.306098 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:55.306108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.308276 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.308281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:55.309584 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:55.309606 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.311388 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.311394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:55.311576 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.311610 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 15:13:55.311724 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 15:13:55.311886 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11386,"thread_start_us":13051,"threads_started":1}
I20260927 15:13:55.312198 18473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.298582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:38865 (local address 127.10.33.67:36271)
0927 15:13:55.299305 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:55.299310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:55.305221 (+ 5911us) server_negotiation.cc:407] Connection header received
0927 15:13:55.305493 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:55.305498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:55.305572 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:55.305659 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:13:55.306256 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.307386 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.309770 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.311148 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.311324 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.311424 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:55.311755 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 15:13:55.311917 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594}
I20260927 15:13:55.312269 17965 catalog_manager.cc:5495] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ac5cf19eb0a4149b1aa89784b1d9f8b (127.10.33.67). New cstate: current_term: 1 leader_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } health_report { overall_health: HEALTHY } } }
I20260927 15:13:55.313062 18404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c232ae01d943f3b59ae37e908404be" candidate_uuid: "82dac9ab7a51481a9bd36d99cac38b25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" is_pre_election: true
I20260927 15:13:55.313709 18055 leader_election.cc:302] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82dac9ab7a51481a9bd36d99cac38b25; no voters: 5b198eed7bc747ef8b132fa94a6203c8, 7ac5cf19eb0a4149b1aa89784b1d9f8b
I20260927 15:13:55.314167 18483 raft_consensus.cc:2726] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 27 15:13:55 dist-test-slave-f5h6 krb5kdc[17920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522032, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.126@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM
I20260927 15:13:55.383865 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.368493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:33393 (local address 127.0.0.1:37648)
0927 15:13:55.369156 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:55.369177 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:55.369348 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:55.369741 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:55.369746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:55.370810 (+ 1064us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:13:55.371153 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:55.371167 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.372507 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.372513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:55.374174 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:55.374191 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.374973 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.374979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:55.375161 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.375892 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:13:55.375916 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:13:55.378614 (+ 2698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:13:55.381401 (+ 2787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:55.381415 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:55.381424 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:55.381804 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:55.382173 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:13:55.382178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:13:55.382182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:13:55.382320 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:13:55.382813 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:13:55.382822 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:13:55.382991 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 15:13:55.383123 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 15:13:55.383398 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":349,"threads_started":1}
I20260927 15:13:55.383884 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.368499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37648 (local address 127.10.33.126:33393)
0927 15:13:55.368767 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:55.368775 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:55.369301 (+ 526us) server_negotiation.cc:407] Connection header received
0927 15:13:55.369527 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:55.369531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:55.369595 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:55.369685 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:55.371357 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.372307 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.374365 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.374777 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.374922 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.378808 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:55.378812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:55.378815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:55.378846 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:55.381263 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:55.381961 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:55.381965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:55.381967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:55.382079 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:55.382428 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:55.382432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:55.382434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:55.382684 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:55.382830 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:55.383235 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 15:13:55.383532 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":125}
I20260927 15:13:55.388878 17965 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:37648:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 15:13:55.389907 17965 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:13:55.396994 18102 tablet_service.cc:1467] Processing CreateTablet for tablet 2b9f352a09bb428ca3316ba5283abeeb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=28a32b1828bf4ef8afee2ba29f2bbd69]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:13:55.397074 18240 tablet_service.cc:1467] Processing CreateTablet for tablet 2b9f352a09bb428ca3316ba5283abeeb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=28a32b1828bf4ef8afee2ba29f2bbd69]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:13:55.397050 18380 tablet_service.cc:1467] Processing CreateTablet for tablet 2b9f352a09bb428ca3316ba5283abeeb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=28a32b1828bf4ef8afee2ba29f2bbd69]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:13:55.397564 18240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b9f352a09bb428ca3316ba5283abeeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:55.397576 18380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b9f352a09bb428ca3316ba5283abeeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:55.397590 18102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b9f352a09bb428ca3316ba5283abeeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:55.405134 18475 tablet_bootstrap.cc:492] T 2b9f352a09bb428ca3316ba5283abeeb P 5b198eed7bc747ef8b132fa94a6203c8: Bootstrap starting.
I20260927 15:13:55.405133 18478 tablet_bootstrap.cc:492] T 2b9f352a09bb428ca3316ba5283abeeb P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Bootstrap starting.
I20260927 15:13:55.405194 18477 tablet_bootstrap.cc:492] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25: Bootstrap starting.
I20260927 15:13:55.406996 18475 tablet_bootstrap.cc:654] T 2b9f352a09bb428ca3316ba5283abeeb P 5b198eed7bc747ef8b132fa94a6203c8: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:55.406991 18478 tablet_bootstrap.cc:654] T 2b9f352a09bb428ca3316ba5283abeeb P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:55.407486 18477 tablet_bootstrap.cc:654] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:55.417161 18478 tablet_bootstrap.cc:492] T 2b9f352a09bb428ca3316ba5283abeeb P 7ac5cf19eb0a4149b1aa89784b1d9f8b: No bootstrap required, opened a new log
I20260927 15:13:55.417352 18478 ts_tablet_manager.cc:1397] T 2b9f352a09bb428ca3316ba5283abeeb P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260927 15:13:55.418215 18478 raft_consensus.cc:348] T 2b9f352a09bb428ca3316ba5283abeeb P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.418434 18478 raft_consensus.cc:374] T 2b9f352a09bb428ca3316ba5283abeeb P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:55.418511 18478 raft_consensus.cc:729] T 2b9f352a09bb428ca3316ba5283abeeb P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac5cf19eb0a4149b1aa89784b1d9f8b, State: Initialized, Role: FOLLOWER
I20260927 15:13:55.418880 18477 tablet_bootstrap.cc:492] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25: No bootstrap required, opened a new log
I20260927 15:13:55.419049 18477 ts_tablet_manager.cc:1397] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.007s
I20260927 15:13:55.419006 18478 consensus_queue.cc:260] T 2b9f352a09bb428ca3316ba5283abeeb P 7ac5cf19eb0a4149b1aa89784b1d9f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.419804 18508 tablet_replica.cc:439] T 2b9f352a09bb428ca3316ba5283abeeb P 7ac5cf19eb0a4149b1aa89784b1d9f8b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } } }
I20260927 15:13:55.419981 18478 ts_tablet_manager.cc:1428] T 2b9f352a09bb428ca3316ba5283abeeb P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 15:13:55.420126 18508 tablet_replica.cc:442] T 2b9f352a09bb428ca3316ba5283abeeb P 7ac5cf19eb0a4149b1aa89784b1d9f8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:13:55.421164 18475 tablet_bootstrap.cc:492] T 2b9f352a09bb428ca3316ba5283abeeb P 5b198eed7bc747ef8b132fa94a6203c8: No bootstrap required, opened a new log
I20260927 15:13:55.421327 18475 ts_tablet_manager.cc:1397] T 2b9f352a09bb428ca3316ba5283abeeb P 5b198eed7bc747ef8b132fa94a6203c8: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.010s
I20260927 15:13:55.422072 18475 raft_consensus.cc:348] T 2b9f352a09bb428ca3316ba5283abeeb P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.422313 18475 raft_consensus.cc:374] T 2b9f352a09bb428ca3316ba5283abeeb P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:55.422389 18475 raft_consensus.cc:729] T 2b9f352a09bb428ca3316ba5283abeeb P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b198eed7bc747ef8b132fa94a6203c8, State: Initialized, Role: FOLLOWER
I20260927 15:13:55.422807 18475 consensus_queue.cc:260] T 2b9f352a09bb428ca3316ba5283abeeb P 5b198eed7bc747ef8b132fa94a6203c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.423062 18477 raft_consensus.cc:348] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.423216 18481 tablet_replica.cc:439] T 2b9f352a09bb428ca3316ba5283abeeb P 5b198eed7bc747ef8b132fa94a6203c8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } } }
I20260927 15:13:55.423370 18477 raft_consensus.cc:374] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:55.423468 18481 tablet_replica.cc:442] T 2b9f352a09bb428ca3316ba5283abeeb P 5b198eed7bc747ef8b132fa94a6203c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:13:55.423573 18477 raft_consensus.cc:729] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82dac9ab7a51481a9bd36d99cac38b25, State: Initialized, Role: FOLLOWER
I20260927 15:13:55.423854 18477 consensus_queue.cc:260] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.424249 18483 tablet_replica.cc:439] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } } }
I20260927 15:13:55.424517 18483 tablet_replica.cc:442] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:13:55.424998 18477 ts_tablet_manager.cc:1428] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20260927 15:13:55.425906 18475 ts_tablet_manager.cc:1428] T 2b9f352a09bb428ca3316ba5283abeeb P 5b198eed7bc747ef8b132fa94a6203c8: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 15:13:55.513101 18483 raft_consensus.cc:482] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:13:55.513319 18483 raft_consensus.cc:504] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.513965 18483 leader_election.cc:288] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b198eed7bc747ef8b132fa94a6203c8 (127.10.33.66:44357), 7ac5cf19eb0a4149b1aa89784b1d9f8b (127.10.33.67:36271)
I20260927 15:13:55.514600 18404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b9f352a09bb428ca3316ba5283abeeb" candidate_uuid: "82dac9ab7a51481a9bd36d99cac38b25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" is_pre_election: true
I20260927 15:13:55.514606 18261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b9f352a09bb428ca3316ba5283abeeb" candidate_uuid: "82dac9ab7a51481a9bd36d99cac38b25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b198eed7bc747ef8b132fa94a6203c8" is_pre_election: true
I20260927 15:13:55.514950 18404 raft_consensus.cc:2454] T 2b9f352a09bb428ca3316ba5283abeeb P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82dac9ab7a51481a9bd36d99cac38b25 in term 0.
I20260927 15:13:55.514994 18261 raft_consensus.cc:2454] T 2b9f352a09bb428ca3316ba5283abeeb P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82dac9ab7a51481a9bd36d99cac38b25 in term 0.
I20260927 15:13:55.515445 18055 leader_election.cc:302] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ac5cf19eb0a4149b1aa89784b1d9f8b, 82dac9ab7a51481a9bd36d99cac38b25; no voters:
I20260927 15:13:55.515683 18483 raft_consensus.cc:2781] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:13:55.515797 18483 raft_consensus.cc:482] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:13:55.515875 18483 raft_consensus.cc:3037] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:55.518196 18483 raft_consensus.cc:504] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.518747 18483 leader_election.cc:288] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25 [CANDIDATE]: Term 1 election: Requested vote from peers 5b198eed7bc747ef8b132fa94a6203c8 (127.10.33.66:44357), 7ac5cf19eb0a4149b1aa89784b1d9f8b (127.10.33.67:36271)
I20260927 15:13:55.519223 18261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b9f352a09bb428ca3316ba5283abeeb" candidate_uuid: "82dac9ab7a51481a9bd36d99cac38b25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b198eed7bc747ef8b132fa94a6203c8"
I20260927 15:13:55.519297 18404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b9f352a09bb428ca3316ba5283abeeb" candidate_uuid: "82dac9ab7a51481a9bd36d99cac38b25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b"
I20260927 15:13:55.519469 18261 raft_consensus.cc:3037] T 2b9f352a09bb428ca3316ba5283abeeb P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:55.519495 18404 raft_consensus.cc:3037] T 2b9f352a09bb428ca3316ba5283abeeb P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:55.521461 18404 raft_consensus.cc:2454] T 2b9f352a09bb428ca3316ba5283abeeb P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82dac9ab7a51481a9bd36d99cac38b25 in term 1.
I20260927 15:13:55.521912 18055 leader_election.cc:302] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ac5cf19eb0a4149b1aa89784b1d9f8b, 82dac9ab7a51481a9bd36d99cac38b25; no voters:
I20260927 15:13:55.522193 18483 raft_consensus.cc:2781] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:55.522640 18483 raft_consensus.cc:686] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25 [term 1 LEADER]: Becoming Leader. State: Replica: 82dac9ab7a51481a9bd36d99cac38b25, State: Running, Role: LEADER
I20260927 15:13:55.522660 18261 raft_consensus.cc:2454] T 2b9f352a09bb428ca3316ba5283abeeb P 5b198eed7bc747ef8b132fa94a6203c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82dac9ab7a51481a9bd36d99cac38b25 in term 1.
I20260927 15:13:55.523115 18483 consensus_queue.cc:237] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.525434 18520 tablet_replica.cc:439] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25: TxnStatusTablet state changed. Reason: New leader 82dac9ab7a51481a9bd36d99cac38b25. Latest consensus state: current_term: 1 leader_uuid: "82dac9ab7a51481a9bd36d99cac38b25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } } }
I20260927 15:13:55.525816 18520 tablet_replica.cc:442] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:13:55.526645 18522 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 15:13:55.526880 18522 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 15:13:55.526985 17965 catalog_manager.cc:5495] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82dac9ab7a51481a9bd36d99cac38b25 (127.10.33.65). New cstate: current_term: 1 leader_uuid: "82dac9ab7a51481a9bd36d99cac38b25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } health_report { overall_health: UNKNOWN } } }
I20260927 15:13:55.558810 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.550792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40256 (local address 127.10.33.65:40973)
0927 15:13:55.550991 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:55.550997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:55.551019 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:13:55.551122 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:55.551127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:55.551238 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:55.551406 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:13:55.551879 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.553063 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.555821 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.556486 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.556636 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.557245 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:13:55.558070 (+ 825us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:13:55.558229 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:55.558353 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 15:13:55.558471 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:13:55.558830 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.550465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:40973 (local address 127.0.0.1:40256)
0927 15:13:55.550728 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:55.550750 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:55.550927 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:55.551478 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:55.551484 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:55.551497 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:13:55.551726 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:55.551740 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.553231 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.553236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:55.555591 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:55.555614 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.556684 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.556691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:55.556936 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.557027 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:13:55.558198 (+ 1171us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:13:55.558223 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:13:55.558319 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 15:13:55.558509 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 15:13:55.563993 18404 raft_consensus.cc:1261] T 2b9f352a09bb428ca3316ba5283abeeb P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 1 FOLLOWER]: Refusing update from remote peer 82dac9ab7a51481a9bd36d99cac38b25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:55.564203 18261 raft_consensus.cc:1261] T 2b9f352a09bb428ca3316ba5283abeeb P 5b198eed7bc747ef8b132fa94a6203c8 [term 1 FOLLOWER]: Refusing update from remote peer 82dac9ab7a51481a9bd36d99cac38b25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:55.564919 18483 consensus_queue.cc:1035] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:55.565336 18520 consensus_queue.cc:1035] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:55.578323 18481 tablet_replica.cc:439] T 2b9f352a09bb428ca3316ba5283abeeb P 5b198eed7bc747ef8b132fa94a6203c8: TxnStatusTablet state changed. Reason: New leader 82dac9ab7a51481a9bd36d99cac38b25. Latest consensus state: current_term: 1 leader_uuid: "82dac9ab7a51481a9bd36d99cac38b25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } } }
I20260927 15:13:55.578651 18481 tablet_replica.cc:442] T 2b9f352a09bb428ca3316ba5283abeeb P 5b198eed7bc747ef8b132fa94a6203c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:13:55.580209 18508 tablet_replica.cc:439] T 2b9f352a09bb428ca3316ba5283abeeb P 7ac5cf19eb0a4149b1aa89784b1d9f8b: TxnStatusTablet state changed. Reason: New leader 82dac9ab7a51481a9bd36d99cac38b25. Latest consensus state: current_term: 1 leader_uuid: "82dac9ab7a51481a9bd36d99cac38b25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } } }
I20260927 15:13:55.580541 18508 tablet_replica.cc:442] T 2b9f352a09bb428ca3316ba5283abeeb P 7ac5cf19eb0a4149b1aa89784b1d9f8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:13:55.584496 18520 tablet_replica.cc:439] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82dac9ab7a51481a9bd36d99cac38b25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } } }
I20260927 15:13:55.584764 18520 tablet_replica.cc:442] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:13:55.588148 18483 tablet_replica.cc:439] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82dac9ab7a51481a9bd36d99cac38b25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } } }
I20260927 15:13:55.588439 18483 tablet_replica.cc:442] T 2b9f352a09bb428ca3316ba5283abeeb P 82dac9ab7a51481a9bd36d99cac38b25: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:13:55.591370 18489 mvcc.cc:205] Tried to move back new op lower bound from 7333978257661681664 to 7333978257506373632. Current Snapshot: MvccSnapshot[applied={T|T < 7333978257661681664}]
I20260927 15:13:55.591677 18481 tablet_replica.cc:439] T 2b9f352a09bb428ca3316ba5283abeeb P 5b198eed7bc747ef8b132fa94a6203c8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82dac9ab7a51481a9bd36d99cac38b25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } } }
I20260927 15:13:55.591919 18481 tablet_replica.cc:442] T 2b9f352a09bb428ca3316ba5283abeeb P 5b198eed7bc747ef8b132fa94a6203c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:13:55.597644 18508 tablet_replica.cc:439] T 2b9f352a09bb428ca3316ba5283abeeb P 7ac5cf19eb0a4149b1aa89784b1d9f8b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82dac9ab7a51481a9bd36d99cac38b25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } } }
I20260927 15:13:55.597975 18508 tablet_replica.cc:442] T 2b9f352a09bb428ca3316ba5283abeeb P 7ac5cf19eb0a4149b1aa89784b1d9f8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:13:55.602972 18473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.592851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33186 (local address 127.10.33.67:36271)
0927 15:13:55.593020 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:55.593026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:55.593044 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:13:55.593144 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:55.593150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:55.593246 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:55.593414 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:13:55.594018 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.595134 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.599997 (+ 4863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.600772 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.600912 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.601528 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:13:55.602316 (+ 788us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:13:55.602474 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:55.602591 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 15:13:55.602687 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:13:55.625624 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.617017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:59427 (local address 127.10.33.65:40973)
0927 15:13:55.617219 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:55.617225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:55.618552 (+ 1327us) server_negotiation.cc:407] Connection header received
0927 15:13:55.619001 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:55.619008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:55.619097 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:55.619156 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:13:55.619907 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.621037 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.623289 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.624079 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.624219 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.624770 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:13:55.624925 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:13:55.625028 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:55.625178 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 15:13:55.625281 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 15:13:55.625800 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.616986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:40973 (local address 127.10.33.67:59427)
0927 15:13:55.618414 (+ 1428us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:55.618438 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:55.618768 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:55.619357 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:55.619362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:55.619375 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:13:55.619710 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:55.619722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.621201 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.621207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:55.623089 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:55.623111 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.624283 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.624290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:55.624481 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.624564 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:13:55.625043 (+ 479us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:13:55.625055 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 15:13:55.625141 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 15:13:55.625329 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1208,"thread_start_us":1035,"threads_started":1}
I20260927 15:13:55.639370 18122 raft_consensus.cc:1261] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25 [term 1 FOLLOWER]: Refusing update from remote peer 7ac5cf19eb0a4149b1aa89784b1d9f8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:55.639685 18260 raft_consensus.cc:1261] T a2c232ae01d943f3b59ae37e908404be P 5b198eed7bc747ef8b132fa94a6203c8 [term 1 FOLLOWER]: Refusing update from remote peer 7ac5cf19eb0a4149b1aa89784b1d9f8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:55.640437 18508 consensus_queue.cc:1035] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:55.640780 18482 consensus_queue.cc:1035] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:55.660282 18491 mvcc.cc:205] Tried to move back new op lower bound from 7333978257971150848 to 7333978256617095168. Current Snapshot: MvccSnapshot[applied={T|T < 7333978257971150848}]
I20260927 15:13:55.706296 18473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.693703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:55569 (local address 127.10.33.67:36271)
0927 15:13:55.693931 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:55.693938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:55.698777 (+ 4839us) server_negotiation.cc:407] Connection header received
0927 15:13:55.698881 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:55.698887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:55.698999 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:55.699111 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:13:55.699929 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.701258 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.703833 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.704536 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.704668 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.705441 (+ 773us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:13:55.705597 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:13:55.705686 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:55.705911 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 15:13:55.705998 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:13:55.706452 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.694095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:36271 (local address 127.10.33.65:55569)
0927 15:13:55.698612 (+ 4517us) negotiation.cc:107] Waiting for socket to connect
0927 15:13:55.698630 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:13:55.698773 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:13:55.699286 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:13:55.699292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:13:55.699307 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:13:55.699721 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:55.699735 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.701474 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.701479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:55.703541 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:13:55.703561 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.704744 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.704750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:13:55.704893 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.704993 (+ 100us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:13:55.705743 (+ 750us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:13:55.705755 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 15:13:55.705841 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 15:13:55.706045 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4316,"thread_start_us":4141,"threads_started":1}
I20260927 15:13:55.748005 17964 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:55.748267 17964 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:55.752252 17964 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a: Sending DeleteTablet for 3 replicas of tablet a2c232ae01d943f3b59ae37e908404be
I20260927 15:13:55.753537 18240 tablet_service.cc:1514] Processing DeleteTablet for tablet a2c232ae01d943f3b59ae37e908404be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:55 UTC) from {username='oryx', principal='oryx/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:45924
I20260927 15:13:55.753626 18102 tablet_service.cc:1514] Processing DeleteTablet for tablet a2c232ae01d943f3b59ae37e908404be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:55 UTC) from {username='oryx', principal='oryx/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:40252
I20260927 15:13:55.754393 18380 tablet_service.cc:1514] Processing DeleteTablet for tablet a2c232ae01d943f3b59ae37e908404be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:55 UTC) from {username='oryx', principal='oryx/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:33180
I20260927 15:13:55.768280 18538 tablet_replica.cc:330] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25: stopping tablet replica
I20260927 15:13:55.768564 18538 raft_consensus.cc:2229] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:55.768862 18538 raft_consensus.cc:2258] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:55.770412 18538 ts_tablet_manager.cc:1905] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:55.779434 18538 ts_tablet_manager.cc:1918] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:13:55.779798 18538 log.cc:1196] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/a2c232ae01d943f3b59ae37e908404be
I20260927 15:13:55.780397 18538 ts_tablet_manager.cc:1939] T a2c232ae01d943f3b59ae37e908404be P 82dac9ab7a51481a9bd36d99cac38b25: Deleting consensus metadata
I20260927 15:13:55.781679 17950 catalog_manager.cc:4841] TS 82dac9ab7a51481a9bd36d99cac38b25 (127.10.33.65:40973): tablet a2c232ae01d943f3b59ae37e908404be (table test-table [id=97f668f418624fb78dbc140bdc71b8d0]) successfully deleted
I20260927 15:13:55.796088 18539 tablet_replica.cc:330] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b: stopping tablet replica
I20260927 15:13:55.796365 18539 raft_consensus.cc:2229] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:13:55.797060 18539 raft_consensus.cc:2258] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:55.797982 18537 tablet_replica.cc:330] T a2c232ae01d943f3b59ae37e908404be P 5b198eed7bc747ef8b132fa94a6203c8: stopping tablet replica
I20260927 15:13:55.798195 18537 raft_consensus.cc:2229] T a2c232ae01d943f3b59ae37e908404be P 5b198eed7bc747ef8b132fa94a6203c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:55.798435 18537 raft_consensus.cc:2258] T a2c232ae01d943f3b59ae37e908404be P 5b198eed7bc747ef8b132fa94a6203c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:55.800325 18539 ts_tablet_manager.cc:1905] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:55.806154 18537 ts_tablet_manager.cc:1905] T a2c232ae01d943f3b59ae37e908404be P 5b198eed7bc747ef8b132fa94a6203c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:55.821727 18537 ts_tablet_manager.cc:1918] T a2c232ae01d943f3b59ae37e908404be P 5b198eed7bc747ef8b132fa94a6203c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:13:55.821911 18537 log.cc:1196] T a2c232ae01d943f3b59ae37e908404be P 5b198eed7bc747ef8b132fa94a6203c8: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/a2c232ae01d943f3b59ae37e908404be
I20260927 15:13:55.822320 18537 ts_tablet_manager.cc:1939] T a2c232ae01d943f3b59ae37e908404be P 5b198eed7bc747ef8b132fa94a6203c8: Deleting consensus metadata
I20260927 15:13:55.823489 17952 catalog_manager.cc:4841] TS 5b198eed7bc747ef8b132fa94a6203c8 (127.10.33.66:44357): tablet a2c232ae01d943f3b59ae37e908404be (table test-table [id=97f668f418624fb78dbc140bdc71b8d0]) successfully deleted
I20260927 15:13:55.826062 18539 ts_tablet_manager.cc:1918] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:13:55.826208 18539 log.cc:1196] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/a2c232ae01d943f3b59ae37e908404be
I20260927 15:13:55.826661 18539 ts_tablet_manager.cc:1939] T a2c232ae01d943f3b59ae37e908404be P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Deleting consensus metadata
I20260927 15:13:55.827893 17952 catalog_manager.cc:4841] TS 7ac5cf19eb0a4149b1aa89784b1d9f8b (127.10.33.67:36271): tablet a2c232ae01d943f3b59ae37e908404be (table test-table [id=97f668f418624fb78dbc140bdc71b8d0]) successfully deleted
Sep 27 15:13:55 dist-test-slave-f5h6 krb5kdc[17920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260927 15:13:55.860838 10373 mini_kdc.cc:317] Time spent kinit for test-admin: real 0.102s user 0.009s sys 0.008s
Sep 27 15:13:55 dist-test-slave-f5h6 krb5kdc[17920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:13:55 dist-test-slave-f5h6 krb5kdc[17920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:13:55.896610 18060 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:13:55.864090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37658 (local address 127.10.33.126:33393)
0927 15:13:55.864277 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:55.864284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:55.881971 (+ 17687us) server_negotiation.cc:407] Connection header received
0927 15:13:55.882245 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:55.882251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:55.882341 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:55.882428 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:55.884297 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.885454 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.887487 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.888317 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.888439 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.896326 (+ 7887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37658: BlockingRecv error: recv got EOF from 127.0.0.1:37658 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 15:13:55 dist-test-slave-f5h6 krb5kdc[17920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM
I20260927 15:13:55.935134 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:55.915973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37670 (local address 127.10.33.126:33393)
0927 15:13:55.916156 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:55.916162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:55.916643 (+ 481us) server_negotiation.cc:407] Connection header received
0927 15:13:55.916913 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:55.916920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:55.917012 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:55.917128 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:55.918932 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.920010 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.921944 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:55.922612 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:55.922756 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:55.928106 (+ 5350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:55.928114 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:55.928120 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:55.928173 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:55.931788 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:55.932844 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:55.932850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:55.932854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:55.933033 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:55.933472 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:55.933477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:55.933481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:55.933845 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:55.934124 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:55.934490 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 15:13:55.934733 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 15:13:55.939265 17964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:13:55.940045 17964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:13:55.948853 18240 tablet_service.cc:1467] Processing CreateTablet for tablet 4ca2a53c28514b52b638eb0be4d6f550 (DEFAULT_TABLE table=test-table [id=81056deb78594f97a5736fb73b274cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:55.948920 18380 tablet_service.cc:1467] Processing CreateTablet for tablet 4ca2a53c28514b52b638eb0be4d6f550 (DEFAULT_TABLE table=test-table [id=81056deb78594f97a5736fb73b274cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:55.949338 18380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ca2a53c28514b52b638eb0be4d6f550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:55.949361 18240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ca2a53c28514b52b638eb0be4d6f550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:55.949350 18102 tablet_service.cc:1467] Processing CreateTablet for tablet 4ca2a53c28514b52b638eb0be4d6f550 (DEFAULT_TABLE table=test-table [id=81056deb78594f97a5736fb73b274cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:55.949792 18102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ca2a53c28514b52b638eb0be4d6f550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:55.961585 18553 tablet_bootstrap.cc:492] T 4ca2a53c28514b52b638eb0be4d6f550 P 5b198eed7bc747ef8b132fa94a6203c8: Bootstrap starting.
I20260927 15:13:55.965409 18553 tablet_bootstrap.cc:654] T 4ca2a53c28514b52b638eb0be4d6f550 P 5b198eed7bc747ef8b132fa94a6203c8: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:55.966130 18554 tablet_bootstrap.cc:492] T 4ca2a53c28514b52b638eb0be4d6f550 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Bootstrap starting.
I20260927 15:13:55.969342 18554 tablet_bootstrap.cc:654] T 4ca2a53c28514b52b638eb0be4d6f550 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:55.972836 18552 tablet_bootstrap.cc:492] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25: Bootstrap starting.
I20260927 15:13:55.975395 18554 tablet_bootstrap.cc:492] T 4ca2a53c28514b52b638eb0be4d6f550 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: No bootstrap required, opened a new log
I20260927 15:13:55.975570 18554 ts_tablet_manager.cc:1397] T 4ca2a53c28514b52b638eb0be4d6f550 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260927 15:13:55.976414 18554 raft_consensus.cc:348] T 4ca2a53c28514b52b638eb0be4d6f550 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.976683 18554 raft_consensus.cc:374] T 4ca2a53c28514b52b638eb0be4d6f550 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:55.976776 18554 raft_consensus.cc:729] T 4ca2a53c28514b52b638eb0be4d6f550 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac5cf19eb0a4149b1aa89784b1d9f8b, State: Initialized, Role: FOLLOWER
I20260927 15:13:55.978693 18552 tablet_bootstrap.cc:654] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:55.979439 18553 tablet_bootstrap.cc:492] T 4ca2a53c28514b52b638eb0be4d6f550 P 5b198eed7bc747ef8b132fa94a6203c8: No bootstrap required, opened a new log
I20260927 15:13:55.979754 18554 consensus_queue.cc:260] T 4ca2a53c28514b52b638eb0be4d6f550 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.980237 18553 ts_tablet_manager.cc:1397] T 4ca2a53c28514b52b638eb0be4d6f550 P 5b198eed7bc747ef8b132fa94a6203c8: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20260927 15:13:55.981142 18553 raft_consensus.cc:348] T 4ca2a53c28514b52b638eb0be4d6f550 P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.983094 18554 ts_tablet_manager.cc:1428] T 4ca2a53c28514b52b638eb0be4d6f550 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260927 15:13:55.985976 18553 raft_consensus.cc:374] T 4ca2a53c28514b52b638eb0be4d6f550 P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:55.986130 18553 raft_consensus.cc:729] T 4ca2a53c28514b52b638eb0be4d6f550 P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b198eed7bc747ef8b132fa94a6203c8, State: Initialized, Role: FOLLOWER
I20260927 15:13:55.986387 18553 consensus_queue.cc:260] T 4ca2a53c28514b52b638eb0be4d6f550 P 5b198eed7bc747ef8b132fa94a6203c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.987102 18553 ts_tablet_manager.cc:1428] T 4ca2a53c28514b52b638eb0be4d6f550 P 5b198eed7bc747ef8b132fa94a6203c8: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20260927 15:13:55.991400 18552 tablet_bootstrap.cc:492] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25: No bootstrap required, opened a new log
I20260927 15:13:55.991606 18552 ts_tablet_manager.cc:1397] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.002s
I20260927 15:13:55.993160 18552 raft_consensus.cc:348] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.993443 18552 raft_consensus.cc:374] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:55.993534 18552 raft_consensus.cc:729] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82dac9ab7a51481a9bd36d99cac38b25, State: Initialized, Role: FOLLOWER
I20260927 15:13:55.993844 18552 consensus_queue.cc:260] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:55.995250 18552 ts_tablet_manager.cc:1428] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260927 15:13:56.218454 18520 raft_consensus.cc:482] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:13:56.218672 18520 raft_consensus.cc:504] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:56.219452 18520 leader_election.cc:288] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b198eed7bc747ef8b132fa94a6203c8 (127.10.33.66:44357), 7ac5cf19eb0a4149b1aa89784b1d9f8b (127.10.33.67:36271)
I20260927 15:13:56.220085 18404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ca2a53c28514b52b638eb0be4d6f550" candidate_uuid: "82dac9ab7a51481a9bd36d99cac38b25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" is_pre_election: true
I20260927 15:13:56.220038 18260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ca2a53c28514b52b638eb0be4d6f550" candidate_uuid: "82dac9ab7a51481a9bd36d99cac38b25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b198eed7bc747ef8b132fa94a6203c8" is_pre_election: true
I20260927 15:13:56.220350 18404 raft_consensus.cc:2454] T 4ca2a53c28514b52b638eb0be4d6f550 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82dac9ab7a51481a9bd36d99cac38b25 in term 0.
I20260927 15:13:56.220371 18260 raft_consensus.cc:2454] T 4ca2a53c28514b52b638eb0be4d6f550 P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82dac9ab7a51481a9bd36d99cac38b25 in term 0.
I20260927 15:13:56.220862 18055 leader_election.cc:302] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ac5cf19eb0a4149b1aa89784b1d9f8b, 82dac9ab7a51481a9bd36d99cac38b25; no voters:
I20260927 15:13:56.221350 18520 raft_consensus.cc:2781] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:13:56.221451 18520 raft_consensus.cc:482] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:13:56.221549 18520 raft_consensus.cc:3037] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:56.224350 18520 raft_consensus.cc:504] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:56.225086 18520 leader_election.cc:288] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25 [CANDIDATE]: Term 1 election: Requested vote from peers 5b198eed7bc747ef8b132fa94a6203c8 (127.10.33.66:44357), 7ac5cf19eb0a4149b1aa89784b1d9f8b (127.10.33.67:36271)
I20260927 15:13:56.225643 18260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ca2a53c28514b52b638eb0be4d6f550" candidate_uuid: "82dac9ab7a51481a9bd36d99cac38b25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b198eed7bc747ef8b132fa94a6203c8"
I20260927 15:13:56.225970 18260 raft_consensus.cc:3037] T 4ca2a53c28514b52b638eb0be4d6f550 P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:56.226068 18404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ca2a53c28514b52b638eb0be4d6f550" candidate_uuid: "82dac9ab7a51481a9bd36d99cac38b25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b"
I20260927 15:13:56.226302 18404 raft_consensus.cc:3037] T 4ca2a53c28514b52b638eb0be4d6f550 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:56.228412 18260 raft_consensus.cc:2454] T 4ca2a53c28514b52b638eb0be4d6f550 P 5b198eed7bc747ef8b132fa94a6203c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82dac9ab7a51481a9bd36d99cac38b25 in term 1.
I20260927 15:13:56.228904 18055 leader_election.cc:302] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b198eed7bc747ef8b132fa94a6203c8, 82dac9ab7a51481a9bd36d99cac38b25; no voters:
I20260927 15:13:56.229182 18520 raft_consensus.cc:2781] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:56.229489 18404 raft_consensus.cc:2454] T 4ca2a53c28514b52b638eb0be4d6f550 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82dac9ab7a51481a9bd36d99cac38b25 in term 1.
I20260927 15:13:56.229558 18520 raft_consensus.cc:686] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25 [term 1 LEADER]: Becoming Leader. State: Replica: 82dac9ab7a51481a9bd36d99cac38b25, State: Running, Role: LEADER
I20260927 15:13:56.229842 18520 consensus_queue.cc:237] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:56.232491 17965 catalog_manager.cc:5495] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82dac9ab7a51481a9bd36d99cac38b25 (127.10.33.65). New cstate: current_term: 1 leader_uuid: "82dac9ab7a51481a9bd36d99cac38b25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } health_report { overall_health: UNKNOWN } } }
I20260927 15:13:56.277066 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:56.271450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.10.33.65:40973)
0927 15:13:56.272004 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:56.272008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:56.272025 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:13:56.272088 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:56.272092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:56.272153 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:56.272266 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:13:56.272759 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:56.273556 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:56.275091 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:56.275529 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:56.275633 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:56.276191 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:13:56.276369 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:13:56.276441 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:56.276648 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 15:13:56.276721 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":355,"threads_started":1}
I20260927 15:13:56.280766 18404 raft_consensus.cc:1261] T 4ca2a53c28514b52b638eb0be4d6f550 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 1 FOLLOWER]: Refusing update from remote peer 82dac9ab7a51481a9bd36d99cac38b25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:56.280797 18260 raft_consensus.cc:1261] T 4ca2a53c28514b52b638eb0be4d6f550 P 5b198eed7bc747ef8b132fa94a6203c8 [term 1 FOLLOWER]: Refusing update from remote peer 82dac9ab7a51481a9bd36d99cac38b25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:56.281359 18520 consensus_queue.cc:1035] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:56.281651 18527 consensus_queue.cc:1035] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:56.286885 18561 mvcc.cc:205] Tried to move back new op lower bound from 7333978260599644160 to 7333978260400111616. Current Snapshot: MvccSnapshot[applied={T|T < 7333978260599644160}]
I20260927 15:13:56.299396 17963 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:56.299610 17963 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:56.303001 17963 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a: Sending DeleteTablet for 3 replicas of tablet 4ca2a53c28514b52b638eb0be4d6f550
I20260927 15:13:56.304101 18240 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ca2a53c28514b52b638eb0be4d6f550 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:56 UTC) from {username='oryx', principal='oryx/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:45924
I20260927 15:13:56.304101 18102 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ca2a53c28514b52b638eb0be4d6f550 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:56 UTC) from {username='oryx', principal='oryx/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:40252
I20260927 15:13:56.304131 18380 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ca2a53c28514b52b638eb0be4d6f550 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:56 UTC) from {username='oryx', principal='oryx/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:33180
I20260927 15:13:56.304390 18537 tablet_replica.cc:330] T 4ca2a53c28514b52b638eb0be4d6f550 P 5b198eed7bc747ef8b132fa94a6203c8: stopping tablet replica
I20260927 15:13:56.304404 18539 tablet_replica.cc:330] T 4ca2a53c28514b52b638eb0be4d6f550 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: stopping tablet replica
I20260927 15:13:56.304600 18539 raft_consensus.cc:2229] T 4ca2a53c28514b52b638eb0be4d6f550 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:56.304602 18537 raft_consensus.cc:2229] T 4ca2a53c28514b52b638eb0be4d6f550 P 5b198eed7bc747ef8b132fa94a6203c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:56.304816 18539 raft_consensus.cc:2258] T 4ca2a53c28514b52b638eb0be4d6f550 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:56.304836 18537 raft_consensus.cc:2258] T 4ca2a53c28514b52b638eb0be4d6f550 P 5b198eed7bc747ef8b132fa94a6203c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:56.306025 18537 ts_tablet_manager.cc:1905] T 4ca2a53c28514b52b638eb0be4d6f550 P 5b198eed7bc747ef8b132fa94a6203c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:56.306607 18539 ts_tablet_manager.cc:1905] T 4ca2a53c28514b52b638eb0be4d6f550 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:56.307137 17963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:13:56.308113 17963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:13:56.305691 18570 tablet_replica.cc:330] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25: stopping tablet replica
I20260927 15:13:56.309598 18570 raft_consensus.cc:2229] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:13:56.310262 18570 raft_consensus.cc:2258] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:56.313992 18537 ts_tablet_manager.cc:1918] T 4ca2a53c28514b52b638eb0be4d6f550 P 5b198eed7bc747ef8b132fa94a6203c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:13:56.314198 18537 log.cc:1196] T 4ca2a53c28514b52b638eb0be4d6f550 P 5b198eed7bc747ef8b132fa94a6203c8: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/4ca2a53c28514b52b638eb0be4d6f550
I20260927 15:13:56.314328 18539 ts_tablet_manager.cc:1918] T 4ca2a53c28514b52b638eb0be4d6f550 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:13:56.314523 18539 log.cc:1196] T 4ca2a53c28514b52b638eb0be4d6f550 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/4ca2a53c28514b52b638eb0be4d6f550
I20260927 15:13:56.314682 18537 ts_tablet_manager.cc:1939] T 4ca2a53c28514b52b638eb0be4d6f550 P 5b198eed7bc747ef8b132fa94a6203c8: Deleting consensus metadata
I20260927 15:13:56.315105 18539 ts_tablet_manager.cc:1939] T 4ca2a53c28514b52b638eb0be4d6f550 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Deleting consensus metadata
I20260927 15:13:56.315945 17952 catalog_manager.cc:4841] TS 5b198eed7bc747ef8b132fa94a6203c8 (127.10.33.66:44357): tablet 4ca2a53c28514b52b638eb0be4d6f550 (table test-table [id=81056deb78594f97a5736fb73b274cc6]) successfully deleted
I20260927 15:13:56.316262 17952 catalog_manager.cc:4841] TS 7ac5cf19eb0a4149b1aa89784b1d9f8b (127.10.33.67:36271): tablet 4ca2a53c28514b52b638eb0be4d6f550 (table test-table [id=81056deb78594f97a5736fb73b274cc6]) successfully deleted
I20260927 15:13:56.317162 18102 tablet_service.cc:1467] Processing CreateTablet for tablet 61c44b90ba264d48bb95f741cce8a6a7 (DEFAULT_TABLE table=test-table [id=28fa74676440443394d62dd8593e96c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:56.317512 18240 tablet_service.cc:1467] Processing CreateTablet for tablet 61c44b90ba264d48bb95f741cce8a6a7 (DEFAULT_TABLE table=test-table [id=28fa74676440443394d62dd8593e96c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:56.317624 18102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c44b90ba264d48bb95f741cce8a6a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:56.317541 18380 tablet_service.cc:1467] Processing CreateTablet for tablet 61c44b90ba264d48bb95f741cce8a6a7 (DEFAULT_TABLE table=test-table [id=28fa74676440443394d62dd8593e96c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:13:56.317879 18240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c44b90ba264d48bb95f741cce8a6a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:56.317972 18380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c44b90ba264d48bb95f741cce8a6a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:13:56.325279 18553 tablet_bootstrap.cc:492] T 61c44b90ba264d48bb95f741cce8a6a7 P 5b198eed7bc747ef8b132fa94a6203c8: Bootstrap starting.
I20260927 15:13:56.326814 18554 tablet_bootstrap.cc:492] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Bootstrap starting.
I20260927 15:13:56.327008 18553 tablet_bootstrap.cc:654] T 61c44b90ba264d48bb95f741cce8a6a7 P 5b198eed7bc747ef8b132fa94a6203c8: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:56.328444 18554 tablet_bootstrap.cc:654] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:56.329679 18570 ts_tablet_manager.cc:1905] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:56.331087 18553 tablet_bootstrap.cc:492] T 61c44b90ba264d48bb95f741cce8a6a7 P 5b198eed7bc747ef8b132fa94a6203c8: No bootstrap required, opened a new log
I20260927 15:13:56.331251 18553 ts_tablet_manager.cc:1397] T 61c44b90ba264d48bb95f741cce8a6a7 P 5b198eed7bc747ef8b132fa94a6203c8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260927 15:13:56.331995 18553 raft_consensus.cc:348] T 61c44b90ba264d48bb95f741cce8a6a7 P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:56.332262 18553 raft_consensus.cc:374] T 61c44b90ba264d48bb95f741cce8a6a7 P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:56.332335 18553 raft_consensus.cc:729] T 61c44b90ba264d48bb95f741cce8a6a7 P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b198eed7bc747ef8b132fa94a6203c8, State: Initialized, Role: FOLLOWER
I20260927 15:13:56.332571 18553 consensus_queue.cc:260] T 61c44b90ba264d48bb95f741cce8a6a7 P 5b198eed7bc747ef8b132fa94a6203c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:56.333117 18553 ts_tablet_manager.cc:1428] T 61c44b90ba264d48bb95f741cce8a6a7 P 5b198eed7bc747ef8b132fa94a6203c8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 15:13:56.337452 18552 tablet_bootstrap.cc:492] T 61c44b90ba264d48bb95f741cce8a6a7 P 82dac9ab7a51481a9bd36d99cac38b25: Bootstrap starting.
I20260927 15:13:56.339178 18552 tablet_bootstrap.cc:654] T 61c44b90ba264d48bb95f741cce8a6a7 P 82dac9ab7a51481a9bd36d99cac38b25: Neither blocks nor log segments found. Creating new log.
I20260927 15:13:56.341125 18552 tablet_bootstrap.cc:492] T 61c44b90ba264d48bb95f741cce8a6a7 P 82dac9ab7a51481a9bd36d99cac38b25: No bootstrap required, opened a new log
I20260927 15:13:56.341287 18552 ts_tablet_manager.cc:1397] T 61c44b90ba264d48bb95f741cce8a6a7 P 82dac9ab7a51481a9bd36d99cac38b25: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 15:13:56.342126 18552 raft_consensus.cc:348] T 61c44b90ba264d48bb95f741cce8a6a7 P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:56.342374 18552 raft_consensus.cc:374] T 61c44b90ba264d48bb95f741cce8a6a7 P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:56.342454 18552 raft_consensus.cc:729] T 61c44b90ba264d48bb95f741cce8a6a7 P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82dac9ab7a51481a9bd36d99cac38b25, State: Initialized, Role: FOLLOWER
I20260927 15:13:56.342515 18570 ts_tablet_manager.cc:1918] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:13:56.342655 18570 log.cc:1196] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/4ca2a53c28514b52b638eb0be4d6f550
I20260927 15:13:56.342715 18552 consensus_queue.cc:260] T 61c44b90ba264d48bb95f741cce8a6a7 P 82dac9ab7a51481a9bd36d99cac38b25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:56.343145 18570 ts_tablet_manager.cc:1939] T 4ca2a53c28514b52b638eb0be4d6f550 P 82dac9ab7a51481a9bd36d99cac38b25: Deleting consensus metadata
I20260927 15:13:56.343703 18552 ts_tablet_manager.cc:1428] T 61c44b90ba264d48bb95f741cce8a6a7 P 82dac9ab7a51481a9bd36d99cac38b25: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:13:56.344274 17950 catalog_manager.cc:4841] TS 82dac9ab7a51481a9bd36d99cac38b25 (127.10.33.65:40973): tablet 4ca2a53c28514b52b638eb0be4d6f550 (table test-table [id=81056deb78594f97a5736fb73b274cc6]) successfully deleted
I20260927 15:13:56.351297 18554 tablet_bootstrap.cc:492] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: No bootstrap required, opened a new log
I20260927 15:13:56.351476 18554 ts_tablet_manager.cc:1397] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.000s
I20260927 15:13:56.352155 18554 raft_consensus.cc:348] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:56.352351 18554 raft_consensus.cc:374] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:13:56.352427 18554 raft_consensus.cc:729] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac5cf19eb0a4149b1aa89784b1d9f8b, State: Initialized, Role: FOLLOWER
I20260927 15:13:56.352666 18554 consensus_queue.cc:260] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:56.353602 18554 ts_tablet_manager.cc:1428] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:13:56.428340 18509 raft_consensus.cc:482] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:13:56.428555 18509 raft_consensus.cc:504] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:56.429240 18509 leader_election.cc:288] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82dac9ab7a51481a9bd36d99cac38b25 (127.10.33.65:40973), 5b198eed7bc747ef8b132fa94a6203c8 (127.10.33.66:44357)
I20260927 15:13:56.429847 18122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c44b90ba264d48bb95f741cce8a6a7" candidate_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82dac9ab7a51481a9bd36d99cac38b25" is_pre_election: true
I20260927 15:13:56.429915 18260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c44b90ba264d48bb95f741cce8a6a7" candidate_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b198eed7bc747ef8b132fa94a6203c8" is_pre_election: true
I20260927 15:13:56.430152 18122 raft_consensus.cc:2454] T 61c44b90ba264d48bb95f741cce8a6a7 P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac5cf19eb0a4149b1aa89784b1d9f8b in term 0.
I20260927 15:13:56.430164 18260 raft_consensus.cc:2454] T 61c44b90ba264d48bb95f741cce8a6a7 P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac5cf19eb0a4149b1aa89784b1d9f8b in term 0.
I20260927 15:13:56.430675 18334 leader_election.cc:302] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5b198eed7bc747ef8b132fa94a6203c8, 7ac5cf19eb0a4149b1aa89784b1d9f8b, 82dac9ab7a51481a9bd36d99cac38b25; no voters:
I20260927 15:13:56.430961 18509 raft_consensus.cc:2781] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:13:56.431102 18509 raft_consensus.cc:482] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:13:56.431191 18509 raft_consensus.cc:3037] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:56.434286 18509 raft_consensus.cc:504] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:56.434998 18509 leader_election.cc:288] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [CANDIDATE]: Term 1 election: Requested vote from peers 82dac9ab7a51481a9bd36d99cac38b25 (127.10.33.65:40973), 5b198eed7bc747ef8b132fa94a6203c8 (127.10.33.66:44357)
I20260927 15:13:56.435657 18260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c44b90ba264d48bb95f741cce8a6a7" candidate_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b198eed7bc747ef8b132fa94a6203c8"
I20260927 15:13:56.435691 18122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c44b90ba264d48bb95f741cce8a6a7" candidate_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82dac9ab7a51481a9bd36d99cac38b25"
I20260927 15:13:56.435925 18260 raft_consensus.cc:3037] T 61c44b90ba264d48bb95f741cce8a6a7 P 5b198eed7bc747ef8b132fa94a6203c8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:56.435952 18122 raft_consensus.cc:3037] T 61c44b90ba264d48bb95f741cce8a6a7 P 82dac9ab7a51481a9bd36d99cac38b25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:13:56.438328 18122 raft_consensus.cc:2454] T 61c44b90ba264d48bb95f741cce8a6a7 P 82dac9ab7a51481a9bd36d99cac38b25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac5cf19eb0a4149b1aa89784b1d9f8b in term 1.
I20260927 15:13:56.438822 18260 raft_consensus.cc:2454] T 61c44b90ba264d48bb95f741cce8a6a7 P 5b198eed7bc747ef8b132fa94a6203c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac5cf19eb0a4149b1aa89784b1d9f8b in term 1.
I20260927 15:13:56.438872 18332 leader_election.cc:302] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ac5cf19eb0a4149b1aa89784b1d9f8b, 82dac9ab7a51481a9bd36d99cac38b25; no voters:
I20260927 15:13:56.439224 18509 raft_consensus.cc:2781] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:13:56.439960 18509 raft_consensus.cc:686] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 1 LEADER]: Becoming Leader. State: Replica: 7ac5cf19eb0a4149b1aa89784b1d9f8b, State: Running, Role: LEADER
I20260927 15:13:56.440245 18509 consensus_queue.cc:237] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } }
I20260927 15:13:56.443365 17963 catalog_manager.cc:5495] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ac5cf19eb0a4149b1aa89784b1d9f8b (127.10.33.67). New cstate: current_term: 1 leader_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac5cf19eb0a4149b1aa89784b1d9f8b" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 36271 } health_report { overall_health: HEALTHY } } }
I20260927 15:13:56.483922 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:56.478009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33188 (local address 127.10.33.67:36271)
0927 15:13:56.478480 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:56.478484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:56.478499 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:13:56.478552 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:56.478556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:56.478614 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:56.478694 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:13:56.479113 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:56.479903 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:56.481898 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:56.482379 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:56.482485 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:56.483029 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:13:56.483159 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:13:56.483232 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:56.483414 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 15:13:56.483485 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":251,"threads_started":1}
I20260927 15:13:56.494813 18260 raft_consensus.cc:1261] T 61c44b90ba264d48bb95f741cce8a6a7 P 5b198eed7bc747ef8b132fa94a6203c8 [term 1 FOLLOWER]: Refusing update from remote peer 7ac5cf19eb0a4149b1aa89784b1d9f8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:56.494804 18122 raft_consensus.cc:1261] T 61c44b90ba264d48bb95f741cce8a6a7 P 82dac9ab7a51481a9bd36d99cac38b25 [term 1 FOLLOWER]: Refusing update from remote peer 7ac5cf19eb0a4149b1aa89784b1d9f8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:13:56.495558 18574 consensus_queue.cc:1035] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "82dac9ab7a51481a9bd36d99cac38b25" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 40973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:56.496162 18509 consensus_queue.cc:1035] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b198eed7bc747ef8b132fa94a6203c8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 44357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:13:56.574674 17963 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:56.574903 17963 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:13:56.577806 17963 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e07a2f8bd2a4e309e5d5c352810526a: Sending DeleteTablet for 3 replicas of tablet 61c44b90ba264d48bb95f741cce8a6a7
I20260927 15:13:56.578683 18102 tablet_service.cc:1514] Processing DeleteTablet for tablet 61c44b90ba264d48bb95f741cce8a6a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:56 UTC) from {username='oryx', principal='oryx/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:40252
I20260927 15:13:56.578794 18380 tablet_service.cc:1514] Processing DeleteTablet for tablet 61c44b90ba264d48bb95f741cce8a6a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:56 UTC) from {username='oryx', principal='oryx/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:33180
I20260927 15:13:56.579087 18539 tablet_replica.cc:330] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: stopping tablet replica
I20260927 15:13:56.579095 18240 tablet_service.cc:1514] Processing DeleteTablet for tablet 61c44b90ba264d48bb95f741cce8a6a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:13:56 UTC) from {username='oryx', principal='oryx/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:45924
I20260927 15:13:56.579278 18539 raft_consensus.cc:2229] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:13:56.579352 18570 tablet_replica.cc:330] T 61c44b90ba264d48bb95f741cce8a6a7 P 82dac9ab7a51481a9bd36d99cac38b25: stopping tablet replica
I20260927 15:13:56.579548 18570 raft_consensus.cc:2229] T 61c44b90ba264d48bb95f741cce8a6a7 P 82dac9ab7a51481a9bd36d99cac38b25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:56.579615 18539 raft_consensus.cc:2258] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:56.579742 18570 raft_consensus.cc:2258] T 61c44b90ba264d48bb95f741cce8a6a7 P 82dac9ab7a51481a9bd36d99cac38b25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:56.580015 18537 tablet_replica.cc:330] T 61c44b90ba264d48bb95f741cce8a6a7 P 5b198eed7bc747ef8b132fa94a6203c8: stopping tablet replica
I20260927 15:13:56.580179 18537 raft_consensus.cc:2229] T 61c44b90ba264d48bb95f741cce8a6a7 P 5b198eed7bc747ef8b132fa94a6203c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:13:56.580358 18537 raft_consensus.cc:2258] T 61c44b90ba264d48bb95f741cce8a6a7 P 5b198eed7bc747ef8b132fa94a6203c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:13:56.580839 18539 ts_tablet_manager.cc:1905] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:56.581449 18537 ts_tablet_manager.cc:1905] T 61c44b90ba264d48bb95f741cce8a6a7 P 5b198eed7bc747ef8b132fa94a6203c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:56.586702 18537 ts_tablet_manager.cc:1918] T 61c44b90ba264d48bb95f741cce8a6a7 P 5b198eed7bc747ef8b132fa94a6203c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:13:56.586915 18537 log.cc:1196] T 61c44b90ba264d48bb95f741cce8a6a7 P 5b198eed7bc747ef8b132fa94a6203c8: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/61c44b90ba264d48bb95f741cce8a6a7
I20260927 15:13:56.586902 18570 ts_tablet_manager.cc:1905] T 61c44b90ba264d48bb95f741cce8a6a7 P 82dac9ab7a51481a9bd36d99cac38b25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:13:56.587401 18537 ts_tablet_manager.cc:1939] T 61c44b90ba264d48bb95f741cce8a6a7 P 5b198eed7bc747ef8b132fa94a6203c8: Deleting consensus metadata
I20260927 15:13:56.587558 18539 ts_tablet_manager.cc:1918] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:13:56.587777 18539 log.cc:1196] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/61c44b90ba264d48bb95f741cce8a6a7
I20260927 15:13:56.588260 18539 ts_tablet_manager.cc:1939] T 61c44b90ba264d48bb95f741cce8a6a7 P 7ac5cf19eb0a4149b1aa89784b1d9f8b: Deleting consensus metadata
I20260927 15:13:56.588717 17952 catalog_manager.cc:4841] TS 5b198eed7bc747ef8b132fa94a6203c8 (127.10.33.66:44357): tablet 61c44b90ba264d48bb95f741cce8a6a7 (table test-table [id=28fa74676440443394d62dd8593e96c0]) successfully deleted
I20260927 15:13:56.590273 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 18036
I20260927 15:13:56.590481 17952 catalog_manager.cc:4841] TS 7ac5cf19eb0a4149b1aa89784b1d9f8b (127.10.33.67:36271): tablet 61c44b90ba264d48bb95f741cce8a6a7 (table test-table [id=28fa74676440443394d62dd8593e96c0]) successfully deleted
I20260927 15:13:56.597464 18570 ts_tablet_manager.cc:1918] T 61c44b90ba264d48bb95f741cce8a6a7 P 82dac9ab7a51481a9bd36d99cac38b25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:13:56.597685 18570 log.cc:1196] T 61c44b90ba264d48bb95f741cce8a6a7 P 82dac9ab7a51481a9bd36d99cac38b25: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/61c44b90ba264d48bb95f741cce8a6a7
I20260927 15:13:56.598174 18570 ts_tablet_manager.cc:1939] T 61c44b90ba264d48bb95f741cce8a6a7 P 82dac9ab7a51481a9bd36d99cac38b25: Deleting consensus metadata
I20260927 15:13:56.599324 17950 catalog_manager.cc:4841] TS 82dac9ab7a51481a9bd36d99cac38b25 (127.10.33.65:40973): tablet 61c44b90ba264d48bb95f741cce8a6a7 (table test-table [id=28fa74676440443394d62dd8593e96c0]) successfully deleted
Sep 27 15:13:56 dist-test-slave-f5h6 krb5kdc[17920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.33.65@KRBTEST.COM
I20260927 15:13:56.616262 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:56.591700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55628 (local address 127.10.33.65:40973)
0927 15:13:56.591908 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:56.591913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:56.598495 (+ 6582us) server_negotiation.cc:407] Connection header received
0927 15:13:56.598683 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:56.598688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:56.598769 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:56.598898 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:56.600430 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:56.601185 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:56.603173 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:56.603640 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:56.603751 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:56.609260 (+ 5509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:56.609291 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:56.609313 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:56.609379 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:56.613060 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:56.614067 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:56.614075 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:56.614077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:56.614236 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:56.614699 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:56.614705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:56.614707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:56.615125 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:56.615395 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:56.615753 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 15:13:56.615954 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 15:13:56.616973 18167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:56.767647 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 18036
I20260927 15:13:56.789206 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 18176
Sep 27 15:13:56 dist-test-slave-f5h6 krb5kdc[17920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.33.66@KRBTEST.COM
I20260927 15:13:56.805440 18587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:56.790131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41862 (local address 127.10.33.66:44357)
0927 15:13:56.790609 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:56.790613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:56.790629 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:13:56.790682 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:56.790685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:56.790748 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:56.790834 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:56.792137 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:56.792846 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:56.794362 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:56.794771 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:56.794864 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:56.800044 (+ 5180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:56.800066 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:56.800085 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:56.800129 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:56.802869 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:56.803624 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:56.803632 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:56.803635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:56.803772 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:56.804050 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:56.804053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:56.804055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:56.804406 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:56.804669 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:56.805010 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 15:13:56.805182 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":243,"threads_started":1}
I20260927 15:13:56.806085 18306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:56.928978 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 18176
I20260927 15:13:56.947206 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 18315
Sep 27 15:13:56 dist-test-slave-f5h6 krb5kdc[17920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.33.67@KRBTEST.COM
I20260927 15:13:56.964274 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:13:56.948032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50730 (local address 127.10.33.67:36271)
0927 15:13:56.948297 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 15:13:56.948304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:13:56.948430 (+ 126us) server_negotiation.cc:407] Connection header received
0927 15:13:56.948619 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:13:56.948623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:13:56.948691 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:13:56.948773 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:13:56.950052 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:56.950759 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:56.951945 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:13:56.952326 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:13:56.952438 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:13:56.957724 (+ 5286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:13:56.957761 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:13:56.957791 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:13:56.957890 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:13:56.961191 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:56.962167 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:56.962175 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:56.962178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:56.962341 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:13:56.962744 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:13:56.962748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:13:56.962751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:13:56.963133 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:13:56.963432 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 15:13:56.963723 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 15:13:56.963957 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260927 15:13:56.964984 18445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:57.086971 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 18315
I20260927 15:13:57.106463 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 17933
I20260927 15:13:57.107580 18005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:13:57.225006 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 17933
2026-09-27T15:13:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6765 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:13:57 dist-test-slave-f5h6 krb5kdc[18593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:13:57 dist-test-slave-f5h6 krb5kdc[18593](info): set up 2 sockets
Sep 27 15:13:57 dist-test-slave-f5h6 krb5kdc[18593](info): commencing operation
krb5kdc: starting...
W20260927 15:13:58.250427 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:13:58 dist-test-slave-f5h6 krb5kdc[18593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:13:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:13:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:13:58.471246 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:36773
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:36527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:36773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:13:58.795157 18606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:13:58.795476 18606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:13:58.795600 18606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:13:58.802944 18606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:13:58.803078 18606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:13:58.803148 18606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:13:58.803221 18606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:13:58.803283 18606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:13:58.814123 18606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:36527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:36773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:36773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:13:58.815472 18606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:13:58.816776 18606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:13:58.829944 18612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:58.830024 18614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:13:58.830049 18611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:13:58.830595 18606 server_base.cc:1034] running on GCE node
I20260927 15:13:58.831368 18606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:13:58.832808 18606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:13:58.833984 18606 hybrid_clock.cc:648] HybridClock initialized: now 1790522038833901 us; error 59 us; skew 500 ppm
I20260927 15:13:58.834324 18606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T15:13:58Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1687 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:13:59 dist-test-slave-f5h6 krb5kdc[18620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:13:59 dist-test-slave-f5h6 krb5kdc[18620](info): set up 2 sockets
Sep 27 15:13:59 dist-test-slave-f5h6 krb5kdc[18620](info): commencing operation
krb5kdc: starting...
W20260927 15:13:59.942561 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.006s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:14:00 dist-test-slave-f5h6 krb5kdc[18620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:14:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:14:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:14:00.165648 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:40655
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:40655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:14:00.488525 18633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:14:00.488864 18633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:14:00.489197 18633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:14:00.497848 18633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:14:00.498023 18633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:14:00.498078 18633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:14:00.498159 18633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:14:00.498208 18633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:14:00.508647 18633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:40655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:40655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:14:00.510025 18633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:14:00.511312 18633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:14:00.522614 18641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:00.522703 18638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:00.522586 18639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:14:00.523923 18633 server_base.cc:1034] running on GCE node
I20260927 15:14:00.524852 18633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:14:00.526438 18633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:14:00.527662 18633 hybrid_clock.cc:648] HybridClock initialized: now 1790522040527613 us; error 44 us; skew 500 ppm
I20260927 15:14:00.527997 18633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:14:00 dist-test-slave-f5h6 krb5kdc[18620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522040, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:14:00.531003 18633 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:14:00.532450 18633 webserver.cc:460] Webserver started at http://127.10.33.126:41251/ using document root <none> and password file <none>
I20260927 15:14:00.533171 18633 fs_manager.cc:359] Metadata directory not provided
I20260927 15:14:00.533283 18633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:14:00.533576 18633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:14:00.536531 18633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "9818899373fd423eaec689795ae2a4e2"
format_stamp: "Formatted at 2026-09-27 15:14:00 on dist-test-slave-f5h6"
server_key: "14c8fcb23bd3a93a63030d1f903dc7f1"
server_key_iv: "663e77f592059af7b5d5686eb1b86537"
server_key_version: "encryptionkey@0"
I20260927 15:14:00.537259 18633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "9818899373fd423eaec689795ae2a4e2"
format_stamp: "Formatted at 2026-09-27 15:14:00 on dist-test-slave-f5h6"
server_key: "14c8fcb23bd3a93a63030d1f903dc7f1"
server_key_iv: "663e77f592059af7b5d5686eb1b86537"
server_key_version: "encryptionkey@0"
I20260927 15:14:00.542251 18633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260927 15:14:00.545612 18648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:00.546923 18633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:14:00.547078 18633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "9818899373fd423eaec689795ae2a4e2"
format_stamp: "Formatted at 2026-09-27 15:14:00 on dist-test-slave-f5h6"
server_key: "14c8fcb23bd3a93a63030d1f903dc7f1"
server_key_iv: "663e77f592059af7b5d5686eb1b86537"
server_key_version: "encryptionkey@0"
I20260927 15:14:00.547252 18633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:14:00.568683 18633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:14:00.572620 18633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:14:00.572872 18633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:14:00.590014 18633 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:40655
I20260927 15:14:00.590046 18699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:40655 every 8 connection(s)
I20260927 15:14:00.591333 18633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:14:00.594921 18700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:14:00.596302 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 18633
I20260927 15:14:00.596525 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:14:00.597211 10373 external_mini_cluster.cc:1428] Setting key 3ee2d69811f9831049292735ba17eddb
I20260927 15:14:00.608289 18700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2: Bootstrap starting.
I20260927 15:14:00.611851 18700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2: Neither blocks nor log segments found. Creating new log.
I20260927 15:14:00.612933 18700 log.cc:824] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:14:00 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:14:00.616475 18700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2: No bootstrap required, opened a new log
I20260927 15:14:00.622758 18703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:00.599389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.10.33.126:40655)
0927 15:14:00.600222 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:00.600242 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:00.600296 (+ 54us) server_negotiation.cc:407] Connection header received
0927 15:14:00.602666 (+ 2370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:00.602698 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:00.603263 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:00.603742 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:00.605155 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:00.607230 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:00.608746 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:00.609737 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:00.609948 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:00.614261 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:00.614270 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:00.614286 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:14:00.614331 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:00.618118 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:00.618832 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:00.618846 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:00.618855 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:00.619056 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:00.619467 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:00.619473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:00.619476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:00.620188 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:00.620469 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:00.620849 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 15:14:00.621291 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":341,"threads_started":1}
I20260927 15:14:00.626228 18700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9818899373fd423eaec689795ae2a4e2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40655 } }
I20260927 15:14:00.626552 18700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:14:00.626624 18700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9818899373fd423eaec689795ae2a4e2, State: Initialized, Role: FOLLOWER
I20260927 15:14:00.627132 18700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9818899373fd423eaec689795ae2a4e2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40655 } }
I20260927 15:14:00.627336 18700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:14:00.627429 18700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:14:00.627539 18700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:14:00.629582 18700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9818899373fd423eaec689795ae2a4e2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40655 } }
I20260927 15:14:00.630065 18700 leader_election.cc:302] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9818899373fd423eaec689795ae2a4e2; no voters:
I20260927 15:14:00.630581 18700 leader_election.cc:288] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:14:00.630713 18705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:14:00.631695 18705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [term 1 LEADER]: Becoming Leader. State: Replica: 9818899373fd423eaec689795ae2a4e2, State: Running, Role: LEADER
I20260927 15:14:00.632259 18705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9818899373fd423eaec689795ae2a4e2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40655 } }
I20260927 15:14:00.632630 18700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:14:00.634961 18706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9818899373fd423eaec689795ae2a4e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9818899373fd423eaec689795ae2a4e2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40655 } } }
I20260927 15:14:00.635341 18706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [sys.catalog]: This master's current role is: LEADER
I20260927 15:14:00.634910 18707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9818899373fd423eaec689795ae2a4e2. Latest consensus state: current_term: 1 leader_uuid: "9818899373fd423eaec689795ae2a4e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9818899373fd423eaec689795ae2a4e2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40655 } } }
I20260927 15:14:00.635785 18707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [sys.catalog]: This master's current role is: LEADER
I20260927 15:14:00.637207 18714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:14:00.641709 18714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:14:00.650364 18714 catalog_manager.cc:1347] Generated new cluster ID: 296fb3782f6d4b61b86b84092839c46a
I20260927 15:14:00.650488 18714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:14:00.668255 18714 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:14:00.669590 18714 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:14:00.681484 18714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2: Generated new TSK 0
I20260927 15:14:00.682439 18714 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.33.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
I20260927 15:14:00.762607 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:40655
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:14:01.096406 18726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:14:01.096714 18726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:14:01.096855 18726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:14:01.104780 18726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:14:01.104903 18726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:14:01.105064 18726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:14:01.115327 18726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:40655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:14:01.116655 18726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:14:01.118002 18726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:14:01.131858 18731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:01.131943 18734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:01.132089 18732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:14:01.133442 18726 server_base.cc:1034] running on GCE node
I20260927 15:14:01.134115 18726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:14:01.135069 18726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:14:01.136304 18726 hybrid_clock.cc:648] HybridClock initialized: now 1790522041136241 us; error 40 us; skew 500 ppm
I20260927 15:14:01.136696 18726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:14:01 dist-test-slave-f5h6 krb5kdc[18620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522041, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:14:01.140599 18726 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:14:01.142318 18726 webserver.cc:460] Webserver started at http://127.10.33.65:44407/ using document root <none> and password file <none>
I20260927 15:14:01.143317 18726 fs_manager.cc:359] Metadata directory not provided
I20260927 15:14:01.143460 18726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:14:01.143816 18726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:14:01.146929 18726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "af098df2d5e94714b0aa219fa5f26f90"
format_stamp: "Formatted at 2026-09-27 15:14:01 on dist-test-slave-f5h6"
server_key: "f94f1ede085f042d7e8c6256af83679e"
server_key_iv: "a19301dc8b5c74a397862ceb5c1fc841"
server_key_version: "encryptionkey@0"
I20260927 15:14:01.147774 18726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "af098df2d5e94714b0aa219fa5f26f90"
format_stamp: "Formatted at 2026-09-27 15:14:01 on dist-test-slave-f5h6"
server_key: "f94f1ede085f042d7e8c6256af83679e"
server_key_iv: "a19301dc8b5c74a397862ceb5c1fc841"
server_key_version: "encryptionkey@0"
I20260927 15:14:01.153532 18726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:14:01.157620 18741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:01.159993 18726 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260927 15:14:01.160200 18726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "af098df2d5e94714b0aa219fa5f26f90"
format_stamp: "Formatted at 2026-09-27 15:14:01 on dist-test-slave-f5h6"
server_key: "f94f1ede085f042d7e8c6256af83679e"
server_key_iv: "a19301dc8b5c74a397862ceb5c1fc841"
server_key_version: "encryptionkey@0"
I20260927 15:14:01.160434 18726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:14:01.178752 18726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:14:01.184454 18726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:14:01.184758 18726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:14:01.185796 18726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:14:01.187258 18726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:14:01.187332 18726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:01.187429 18726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:14:01.187505 18726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:01.223147 18726 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:37951
I20260927 15:14:01.223176 18853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:37951 every 8 connection(s)
I20260927 15:14:01.224470 18726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:14:01.234751 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 18726
I20260927 15:14:01.234966 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:14:01.235680 10373 external_mini_cluster.cc:1428] Setting key d36534f422752e0754a6487c85a94db4
Sep 27 15:14:01 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522041, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:14:01.244688 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:01.227046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:45941 (local address 127.10.33.126:40655)
0927 15:14:01.227463 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:01.227468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:01.228246 (+ 778us) server_negotiation.cc:407] Connection header received
0927 15:14:01.230054 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:01.230060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:01.230138 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:01.230226 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:01.231704 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:01.232430 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:01.234079 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:01.234461 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:01.234573 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:01.238431 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:01.238438 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:01.238442 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:14:01.238476 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:01.241418 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:01.242266 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:01.242271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:01.242275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:01.242406 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:01.242881 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:01.242885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:01.242888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:01.243215 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:01.243349 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:01.244121 (+ 772us) server_negotiation.cc:299] Negotiation successful
0927 15:14:01.244342 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":224,"threads_started":1}
I20260927 15:14:01.245615 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:01.227436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.65:45941)
0927 15:14:01.228095 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:01.228130 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:01.229752 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:01.230462 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:01.230471 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:01.230963 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:01.231539 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:01.231557 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:01.232603 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:01.232607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:01.233937 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:01.233951 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:01.234678 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:01.234686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:01.234922 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:01.235677 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:01.235699 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:01.238207 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:14:01.241595 (+ 3388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:14:01.241602 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:14:01.241619 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:14:01.242145 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:14:01.242552 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:14:01.242556 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:14:01.242558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:14:01.242770 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:14:01.243345 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:14:01.243352 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:14:01.243748 (+ 396us) client_negotiation.cc:769] Sending connection context
0927 15:14:01.243985 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 15:14:01.244411 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":250,"threads_started":1}
I20260927 15:14:01.247206 18854 heartbeater.cc:347] Connected to a master server at 127.10.33.126:40655
I20260927 15:14:01.247503 18854 heartbeater.cc:464] Registering TS with master...
I20260927 15:14:01.248219 18854 heartbeater.cc:511] Master 127.10.33.126:40655 requested a full tablet report, sending...
I20260927 15:14:01.250332 18665 ts_manager.cc:194] Registered new tserver with Master: af098df2d5e94714b0aa219fa5f26f90 (127.10.33.65:37951)
I20260927 15:14:01.251899 18665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:45941
WARNING: no policy specified for kudu/127.10.33.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
I20260927 15:14:01.286944 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:40655
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:14:01.610263 18860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:14:01.610571 18860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:14:01.610689 18860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:14:01.618148 18860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:14:01.618278 18860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:14:01.618420 18860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:14:01.628813 18860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:40655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:14:01.630405 18860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:14:01.631762 18860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:14:01.645778 18866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:01.645884 18865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:01.646139 18868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:14:01.646504 18860 server_base.cc:1034] running on GCE node
I20260927 15:14:01.647135 18860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:14:01.648233 18860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:14:01.649530 18860 hybrid_clock.cc:648] HybridClock initialized: now 1790522041649445 us; error 38 us; skew 500 ppm
I20260927 15:14:01.649896 18860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:14:01 dist-test-slave-f5h6 krb5kdc[18620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522041, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:14:01.653252 18860 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:14:01.654788 18860 webserver.cc:460] Webserver started at http://127.10.33.66:36795/ using document root <none> and password file <none>
I20260927 15:14:01.655494 18860 fs_manager.cc:359] Metadata directory not provided
I20260927 15:14:01.655612 18860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:14:01.655889 18860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:14:01.659170 18860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "75e53bd764994e5aa00236e19c89cfac"
format_stamp: "Formatted at 2026-09-27 15:14:01 on dist-test-slave-f5h6"
server_key: "66be934679578e45910444878e6b61bc"
server_key_iv: "bf6bac56462b2900a54a36c13ecf3ffe"
server_key_version: "encryptionkey@0"
I20260927 15:14:01.659948 18860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "75e53bd764994e5aa00236e19c89cfac"
format_stamp: "Formatted at 2026-09-27 15:14:01 on dist-test-slave-f5h6"
server_key: "66be934679578e45910444878e6b61bc"
server_key_iv: "bf6bac56462b2900a54a36c13ecf3ffe"
server_key_version: "encryptionkey@0"
I20260927 15:14:01.665390 18860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 15:14:01.669219 18875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:01.670683 18860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:14:01.670861 18860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "75e53bd764994e5aa00236e19c89cfac"
format_stamp: "Formatted at 2026-09-27 15:14:01 on dist-test-slave-f5h6"
server_key: "66be934679578e45910444878e6b61bc"
server_key_iv: "bf6bac56462b2900a54a36c13ecf3ffe"
server_key_version: "encryptionkey@0"
I20260927 15:14:01.671072 18860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:14:01.683538 18860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:14:01.687737 18860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:14:01.688017 18860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:14:01.688931 18860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:14:01.690376 18860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:14:01.690469 18860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:01.690580 18860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:14:01.690636 18860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:01.725586 18860 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:36151
I20260927 15:14:01.725596 18987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:36151 every 8 connection(s)
I20260927 15:14:01.727051 18860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:14:01.728279 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 18860
I20260927 15:14:01.728534 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:14:01.729049 10373 external_mini_cluster.cc:1428] Setting key 4c94b96c537da46fbb2e6eada4414b96
Sep 27 15:14:01 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522041, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:14:01.750384 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:01.730173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:56423 (local address 127.10.33.126:40655)
0927 15:14:01.730367 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:01.730374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:01.731731 (+ 1357us) server_negotiation.cc:407] Connection header received
0927 15:14:01.734377 (+ 2646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:01.734382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:01.734509 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:01.734606 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:01.737015 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:01.737781 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:01.740125 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:01.740596 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:01.740718 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:01.744252 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:01.744258 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:01.744261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:14:01.744294 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:01.746945 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:01.747813 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:01.747820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:01.747824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:01.748003 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:01.748445 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:01.748450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:01.748454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:01.748817 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:01.748990 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:01.749696 (+ 706us) server_negotiation.cc:299] Negotiation successful
0927 15:14:01.750007 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:14:01.751358 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:01.730663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.66:56423)
0927 15:14:01.731514 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:01.731564 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:01.734008 (+ 2444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:01.734905 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:01.734929 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:01.735656 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:01.736717 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:01.736751 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:01.737978 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:01.737983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:01.739939 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:01.739956 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:01.740842 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:01.740849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:01.741095 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:01.741774 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:01.741798 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:01.743992 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:14:01.747129 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:14:01.747136 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:14:01.747153 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:14:01.747650 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:14:01.748126 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:14:01.748129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:14:01.748132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:14:01.748327 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:14:01.748954 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:14:01.748961 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:14:01.749314 (+ 353us) client_negotiation.cc:769] Sending connection context
0927 15:14:01.749558 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 15:14:01.750060 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":382,"threads_started":1}
WARNING: no policy specified for kudu/127.10.33.67@KRBTEST.COM; defaulting to no policy
I20260927 15:14:01.752974 18988 heartbeater.cc:347] Connected to a master server at 127.10.33.126:40655
I20260927 15:14:01.753309 18988 heartbeater.cc:464] Registering TS with master...
I20260927 15:14:01.754105 18988 heartbeater.cc:511] Master 127.10.33.126:40655 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.67@KRBTEST.COM" created.
I20260927 15:14:01.755686 18665 ts_manager.cc:194] Registered new tserver with Master: 75e53bd764994e5aa00236e19c89cfac (127.10.33.66:36151)
I20260927 15:14:01.756554 18665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:56423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
I20260927 15:14:01.785753 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:40655
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:14:02.123649 18993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:14:02.123935 18993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:14:02.124063 18993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:14:02.131570 18993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:14:02.131696 18993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:14:02.131842 18993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:14:02.142441 18993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:40655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:14:02.143776 18993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:14:02.145074 18993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:14:02.159283 19001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:02.159328 18998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:02.159346 18999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:14:02.160267 18993 server_base.cc:1034] running on GCE node
I20260927 15:14:02.161118 18993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:14:02.162650 18993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:14:02.163913 18993 hybrid_clock.cc:648] HybridClock initialized: now 1790522042163842 us; error 58 us; skew 500 ppm
I20260927 15:14:02.164263 18993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:14:02 dist-test-slave-f5h6 krb5kdc[18620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522042, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:14:02.167685 18993 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:14:02.169142 18993 webserver.cc:460] Webserver started at http://127.10.33.67:41421/ using document root <none> and password file <none>
I20260927 15:14:02.169798 18993 fs_manager.cc:359] Metadata directory not provided
I20260927 15:14:02.169943 18993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:14:02.170274 18993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:14:02.173079 18993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "21c909b002c44d81871578de6c430be5"
format_stamp: "Formatted at 2026-09-27 15:14:02 on dist-test-slave-f5h6"
server_key: "067a06750d1cbf8df502e5058b9b9a07"
server_key_iv: "e383f4bfecd29b85c4fb9d52bddc0819"
server_key_version: "encryptionkey@0"
I20260927 15:14:02.173808 18993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "21c909b002c44d81871578de6c430be5"
format_stamp: "Formatted at 2026-09-27 15:14:02 on dist-test-slave-f5h6"
server_key: "067a06750d1cbf8df502e5058b9b9a07"
server_key_iv: "e383f4bfecd29b85c4fb9d52bddc0819"
server_key_version: "encryptionkey@0"
I20260927 15:14:02.178817 18993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 15:14:02.182389 19008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:02.183949 18993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 15:14:02.184159 18993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "21c909b002c44d81871578de6c430be5"
format_stamp: "Formatted at 2026-09-27 15:14:02 on dist-test-slave-f5h6"
server_key: "067a06750d1cbf8df502e5058b9b9a07"
server_key_iv: "e383f4bfecd29b85c4fb9d52bddc0819"
server_key_version: "encryptionkey@0"
I20260927 15:14:02.184361 18993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:14:02.219652 18993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:14:02.224061 18993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:14:02.224319 18993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:14:02.225261 18993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:14:02.226624 18993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:14:02.226715 18993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:02.226842 18993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:14:02.226892 18993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:02.255013 18854 heartbeater.cc:503] Master 127.10.33.126:40655 was elected leader, sending a full tablet report...
I20260927 15:14:02.257066 18993 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:46843
I20260927 15:14:02.257138 19120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:46843 every 8 connection(s)
I20260927 15:14:02.258303 18993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:14:02.259410 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 18993
I20260927 15:14:02.259600 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:14:02.260165 10373 external_mini_cluster.cc:1428] Setting key 2c502c5f273695a7df28cf2fa1b1b02d
Sep 27 15:14:02 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522042, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:14:02.277664 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:02.260785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:35997 (local address 127.10.33.126:40655)
0927 15:14:02.261201 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:02.261207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:02.262376 (+ 1169us) server_negotiation.cc:407] Connection header received
0927 15:14:02.264322 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:02.264326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:02.264387 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:02.264482 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:02.266105 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:02.266882 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:02.268424 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:02.268837 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:02.268938 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:02.272054 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:02.272059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:02.272062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:14:02.272093 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:02.274770 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:02.275559 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:02.275563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:02.275566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:02.275680 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:02.276060 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:02.276063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:02.276065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:02.276308 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:02.276432 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:02.277087 (+ 655us) server_negotiation.cc:299] Negotiation successful
0927 15:14:02.277290 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":195,"threads_started":1}
I20260927 15:14:02.278648 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:02.261437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.67:35997)
0927 15:14:02.262243 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:02.262284 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:02.264043 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:02.264679 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:02.264688 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:02.265203 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:02.265936 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:02.265957 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:02.267040 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:02.267044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:02.268282 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:02.268294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:02.269056 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:02.269061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:02.269250 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:02.269829 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:02.269876 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:02.271873 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:14:02.274889 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:14:02.274897 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:14:02.274910 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:14:02.275440 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:14:02.275774 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:14:02.275779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:14:02.275782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:14:02.275945 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:14:02.276422 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:14:02.276429 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:14:02.276730 (+ 301us) client_negotiation.cc:769] Sending connection context
0927 15:14:02.276978 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 15:14:02.277460 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":320,"threads_started":1}
I20260927 15:14:02.279838 19121 heartbeater.cc:347] Connected to a master server at 127.10.33.126:40655
I20260927 15:14:02.280102 19121 heartbeater.cc:464] Registering TS with master...
I20260927 15:14:02.280750 19121 heartbeater.cc:511] Master 127.10.33.126:40655 requested a full tablet report, sending...
I20260927 15:14:02.282131 18665 ts_manager.cc:194] Registered new tserver with Master: 21c909b002c44d81871578de6c430be5 (127.10.33.67:46843)
I20260927 15:14:02.282922 18665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:35997
I20260927 15:14:02.287842 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.33.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.33.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.33.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.65.keytab.
Entry for principal oryx/127.10.33.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.65.keytab.
WARNING: no policy specified for oryx/127.10.33.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.33.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.33.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.66.keytab.
Entry for principal oryx/127.10.33.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.66.keytab.
WARNING: no policy specified for oryx/127.10.33.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.33.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.33.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.67.keytab.
Entry for principal oryx/127.10.33.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.67.keytab.
I20260927 15:14:02.426069 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 18726
Sep 27 15:14:02 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.65@KRBTEST.COM
I20260927 15:14:02.442912 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:02.427421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37928 (local address 127.10.33.65:37951)
0927 15:14:02.428081 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:02.428091 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:02.428125 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:14:02.428194 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:02.428201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:02.428407 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:02.428555 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:02.429970 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:02.430975 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:02.432207 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:02.432774 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:02.432890 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:02.437208 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:02.437223 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:02.437241 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:14:02.437290 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:02.440136 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:02.440845 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:02.440853 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:02.440857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:02.440984 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:02.441332 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:02.441336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:02.441338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:02.441706 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:02.442005 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:02.442304 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 15:14:02.442572 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":284,"threads_started":1}
I20260927 15:14:02.444308 18849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:14:02.526275 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 18726
I20260927 15:14:02.539151 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 18860
Sep 27 15:14:02 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.66@KRBTEST.COM
I20260927 15:14:02.556574 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:02.539971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32960 (local address 127.10.33.66:36151)
0927 15:14:02.540529 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:02.540536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:02.540563 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:14:02.540619 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:02.540626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:02.540836 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:02.540960 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:02.542320 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:02.543455 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:02.544714 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:02.545307 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:02.545426 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:02.549926 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:02.549944 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:02.549961 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:14:02.550012 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:02.553326 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:02.554238 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:02.554247 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:02.554250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:02.554386 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:02.554915 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:02.554920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:02.554923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:02.555341 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:02.555616 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:02.556012 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 15:14:02.556231 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":278,"threads_started":1}
I20260927 15:14:02.557798 18983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:14:02.640228 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 18860
I20260927 15:14:02.653117 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 18993
Sep 27 15:14:02 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.67@KRBTEST.COM
I20260927 15:14:02.670984 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:02.654122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58508 (local address 127.10.33.67:46843)
0927 15:14:02.654720 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:02.654727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:02.654756 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:14:02.654817 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:02.654823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:02.655050 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:02.655182 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:02.656588 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:02.657605 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:02.659092 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:02.659806 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:02.659926 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:02.664327 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:02.664343 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:02.664359 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:14:02.664400 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:02.667673 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:02.668484 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:02.668492 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:02.668494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:02.668725 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:02.669192 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:02.669196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:02.669198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:02.669696 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:02.670023 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:02.670330 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 15:14:02.670634 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":300,"threads_started":1}
I20260927 15:14:02.672294 19116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:14:02.755551 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 18993
I20260927 15:14:02.768693 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 18633
I20260927 15:14:02.770081 18695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:14:02.871321 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 18633
I20260927 15:14:02.889940 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:40655
--webserver_interface=127.10.33.126
--webserver_port=41251
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:40655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:14:03.234002 19139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:14:03.234339 19139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:14:03.234463 19139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:14:03.241849 19139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:14:03.242023 19139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:14:03.242105 19139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:14:03.242177 19139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:14:03.242234 19139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:14:03.253027 19139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:40655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:40655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=41251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:14:03.254515 19139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:14:03.255841 19139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:14:03.268249 19144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:03.268370 19147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:03.268290 19145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:14:03.269655 19139 server_base.cc:1034] running on GCE node
I20260927 15:14:03.270539 19139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:14:03.272063 19139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:14:03.273291 19139 hybrid_clock.cc:648] HybridClock initialized: now 1790522043273242 us; error 38 us; skew 500 ppm
I20260927 15:14:03.273677 19139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:14:03 dist-test-slave-f5h6 krb5kdc[18620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522043, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:14:03.277118 19139 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:14:03.278779 19139 webserver.cc:460] Webserver started at http://127.10.33.126:41251/ using document root <none> and password file <none>
I20260927 15:14:03.279481 19139 fs_manager.cc:359] Metadata directory not provided
I20260927 15:14:03.279598 19139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:14:03.285174 19139 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:14:03.288893 19154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:03.290779 19139 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:14:03.290961 19139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "9818899373fd423eaec689795ae2a4e2"
format_stamp: "Formatted at 2026-09-27 15:14:00 on dist-test-slave-f5h6"
server_key: "14c8fcb23bd3a93a63030d1f903dc7f1"
server_key_iv: "663e77f592059af7b5d5686eb1b86537"
server_key_version: "encryptionkey@0"
I20260927 15:14:03.291983 19139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:14:03.310699 19139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:14:03.315318 19139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:14:03.315590 19139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:14:03.335227 19139 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:40655
I20260927 15:14:03.335251 19205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:40655 every 8 connection(s)
I20260927 15:14:03.336818 19139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:14:03.343055 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 19139
I20260927 15:14:03.344525 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:37951
--local_ip_for_outbound_sockets=127.10.33.65
--tserver_master_addrs=127.10.33.126:40655
--webserver_port=44407
--webserver_interface=127.10.33.65
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.33.65
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:14:03.345371 19206 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:14:03.350076 19206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2: Bootstrap starting.
I20260927 15:14:03.369671 19206 log.cc:824] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2: Log is configured to *not* fsync() on all Append() calls
I20260927 15:14:03.378893 19206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:14:03.379390 19206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2: Bootstrap complete.
I20260927 15:14:03.388626 19206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9818899373fd423eaec689795ae2a4e2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40655 } }
I20260927 15:14:03.389451 19206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9818899373fd423eaec689795ae2a4e2, State: Initialized, Role: FOLLOWER
I20260927 15:14:03.390102 19206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9818899373fd423eaec689795ae2a4e2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40655 } }
I20260927 15:14:03.390323 19206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:14:03.390429 19206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:14:03.390599 19206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:14:03.394526 19206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9818899373fd423eaec689795ae2a4e2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40655 } }
I20260927 15:14:03.395076 19206 leader_election.cc:302] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9818899373fd423eaec689795ae2a4e2; no voters:
I20260927 15:14:03.395691 19206 leader_election.cc:288] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:14:03.395834 19210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:14:03.397065 19210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [term 2 LEADER]: Becoming Leader. State: Replica: 9818899373fd423eaec689795ae2a4e2, State: Running, Role: LEADER
I20260927 15:14:03.397675 19210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9818899373fd423eaec689795ae2a4e2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40655 } }
I20260927 15:14:03.398329 19206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:14:03.401533 19211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9818899373fd423eaec689795ae2a4e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9818899373fd423eaec689795ae2a4e2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40655 } } }
I20260927 15:14:03.402068 19211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [sys.catalog]: This master's current role is: LEADER
I20260927 15:14:03.402180 19212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9818899373fd423eaec689795ae2a4e2. Latest consensus state: current_term: 2 leader_uuid: "9818899373fd423eaec689795ae2a4e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9818899373fd423eaec689795ae2a4e2" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40655 } } }
I20260927 15:14:03.402588 19212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2 [sys.catalog]: This master's current role is: LEADER
I20260927 15:14:03.415692 19226 catalog_manager.cc:1259] Loaded cluster ID: 296fb3782f6d4b61b86b84092839c46a
I20260927 15:14:03.415827 19226 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2: loading cluster ID for follower catalog manager: success
I20260927 15:14:03.419832 19226 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2: acquiring CA information for follower catalog manager: success
I20260927 15:14:03.422405 19226 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:14:03.422688 19216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:14:03.425174 19216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:14:03.426317 19216 catalog_manager.cc:1259] Loaded cluster ID: 296fb3782f6d4b61b86b84092839c46a
I20260927 15:14:03.426422 19216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:14:03.428099 19216 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:14:03.429188 19216 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9818899373fd423eaec689795ae2a4e2: Loaded TSK: 0
I20260927 15:14:03.430235 19216 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:14:03.690944 19208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:14:03.691331 19208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:14:03.691483 19208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:14:03.699380 19208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:14:03.699515 19208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:14:03.699668 19208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:14:03.710244 19208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.65.keytab
--principal=oryx/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:37951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=44407
--tserver_master_addrs=127.10.33.126:40655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:14:03.711910 19208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:14:03.713259 19208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:14:03.728590 19232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:03.728749 19233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:03.729314 19235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:14:03.729915 19208 server_base.cc:1034] running on GCE node
I20260927 15:14:03.730589 19208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:14:03.731568 19208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:14:03.732808 19208 hybrid_clock.cc:648] HybridClock initialized: now 1790522043732735 us; error 58 us; skew 500 ppm
I20260927 15:14:03.733176 19208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:14:03 dist-test-slave-f5h6 krb5kdc[18620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522043, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:14:03.736451 19208 init.cc:377] Logged in from keytab as oryx/127.10.33.65@KRBTEST.COM (short username oryx)
I20260927 15:14:03.738044 19208 webserver.cc:460] Webserver started at http://127.10.33.65:44407/ using document root <none> and password file <none>
I20260927 15:14:03.738749 19208 fs_manager.cc:359] Metadata directory not provided
I20260927 15:14:03.738879 19208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:14:03.744753 19208 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:14:03.747941 19242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:03.749704 19208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:14:03.749943 19208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "af098df2d5e94714b0aa219fa5f26f90"
format_stamp: "Formatted at 2026-09-27 15:14:01 on dist-test-slave-f5h6"
server_key: "f94f1ede085f042d7e8c6256af83679e"
server_key_iv: "a19301dc8b5c74a397862ceb5c1fc841"
server_key_version: "encryptionkey@0"
I20260927 15:14:03.750973 19208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:14:03.769893 19208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:14:03.774536 19208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:14:03.774819 19208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:14:03.775882 19208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:14:03.777283 19208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:14:03.777376 19208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:03.777448 19208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:14:03.777504 19208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:03.814316 19208 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:37951
I20260927 15:14:03.814370 19354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:37951 every 8 connection(s)
I20260927 15:14:03.815788 19208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:14:03.821462 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 19208
I20260927 15:14:03.822106 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:36151
--local_ip_for_outbound_sockets=127.10.33.66
--tserver_master_addrs=127.10.33.126:40655
--webserver_port=36795
--webserver_interface=127.10.33.66
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.33.66
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:14:03 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:03 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:14:04.187111 19359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:14:04.187407 19359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:14:04.187508 19359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:14:04.195305 19359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:14:04.195448 19359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:14:04.195595 19359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:14:04.206283 19359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.66.keytab
--principal=oryx/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:36151
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=36795
--tserver_master_addrs=127.10.33.126:40655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:14:04.207686 19359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:14:04.209592 19359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:14:04.230286 19368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:04.230213 19366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:04.230572 19365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:14:04.231832 19359 server_base.cc:1034] running on GCE node
I20260927 15:14:04.232539 19359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:14:04.233616 19359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:14:04.235126 19359 hybrid_clock.cc:648] HybridClock initialized: now 1790522044235031 us; error 89 us; skew 500 ppm
I20260927 15:14:04.235497 19359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:14:04 dist-test-slave-f5h6 krb5kdc[18620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522044, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:14:04.252524 19359 init.cc:377] Logged in from keytab as oryx/127.10.33.66@KRBTEST.COM (short username oryx)
I20260927 15:14:04.254755 19359 webserver.cc:460] Webserver started at http://127.10.33.66:36795/ using document root <none> and password file <none>
I20260927 15:14:04.255940 19359 fs_manager.cc:359] Metadata directory not provided
I20260927 15:14:04.256160 19359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:14:04.264695 19359 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:14:04.268776 19375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:04.270798 19359 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:14:04.271027 19359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "75e53bd764994e5aa00236e19c89cfac"
format_stamp: "Formatted at 2026-09-27 15:14:01 on dist-test-slave-f5h6"
server_key: "66be934679578e45910444878e6b61bc"
server_key_iv: "bf6bac56462b2900a54a36c13ecf3ffe"
server_key_version: "encryptionkey@0"
I20260927 15:14:04.272769 19359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:14:04.306324 19359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:14:04.314188 19359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:14:04.315034 19359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:14:04.316780 19359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:14:04.319214 19359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:14:04.319396 19359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:04.319553 19359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:14:04.319643 19359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:04.364274 19359 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:36151
I20260927 15:14:04.364315 19487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:36151 every 8 connection(s)
I20260927 15:14:04.365724 19359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:14:04.367602 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 19359
I20260927 15:14:04.368371 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:46843
--local_ip_for_outbound_sockets=127.10.33.67
--tserver_master_addrs=127.10.33.126:40655
--webserver_port=41421
--webserver_interface=127.10.33.67
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.33.67
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:14:04 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:04 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:14:04.861330 19358 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.031s user 0.004s sys 0.008s
W20260927 15:14:04.863407 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:03.818829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:50325 (local address 127.10.33.126:40655)
0927 15:14:03.819633 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:03.819649 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:03.820169 (+ 520us) server_negotiation.cc:407] Connection header received
0927 15:14:03.822807 (+ 2638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:03.822821 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:03.823182 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:03.823532 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:03.825101 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:03.826412 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:03.827974 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:03.828620 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:03.828802 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:04.862664 (+1033862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.65:50325: BlockingRecv error: recv got EOF from 127.10.33.65:50325 (error 108)
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":319,"threads_started":1}
I20260927 15:14:04.863409 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:03.818878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.65:50325)
0927 15:14:03.819991 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:03.820036 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:03.822174 (+ 2138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:03.823703 (+ 1529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:03.823715 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:03.824338 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:03.824887 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:03.824906 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:03.826592 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:03.826596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:03.827821 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:03.827835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:03.828857 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:03.828864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:03.829183 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:03.829849 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:03.829899 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:04.862289 (+1032390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":873,"thread_start_us":679,"threads_started":1}
W20260927 15:14:04.864501 19355 heartbeater.cc:644] Failed to heartbeat to 127.10.33.126:40655 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.33.126:40655: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Sep 27 15:14:04 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:04 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:04.875104 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:04.865190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.65:60231)
0927 15:14:04.866156 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:04.866180 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:04.866325 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:04.866791 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:04.866796 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:04.867168 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:04.867341 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:04.867350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:04.868523 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:04.868527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:04.869702 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:04.869717 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:04.870489 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:04.870496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:04.870675 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:04.871335 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:04.871354 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:04.874789 (+ 3435us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":1559,"threads_started":1}
W20260927 15:14:04.875258 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:04.865586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:60231 (local address 127.10.33.126:40655)
0927 15:14:04.865732 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:04.865738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:04.866248 (+ 510us) server_negotiation.cc:407] Connection header received
0927 15:14:04.866450 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:04.866456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:04.866562 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:04.866653 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:04.867487 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:04.868347 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:04.869890 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:04.870311 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:04.870427 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:04.875038 (+ 4611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.65:60231: BlockingRecv error: recv got EOF from 127.10.33.65:60231 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 27 15:14:04 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:04 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:04.884799 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:04.876062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.65:54501)
0927 15:14:04.876344 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:04.876362 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:04.876474 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:04.876791 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:04.876795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:04.877129 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:04.877282 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:04.877291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:04.878369 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:04.878375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:04.879444 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:04.879457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:04.880254 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:04.880260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:04.880412 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:04.881052 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:04.881079 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:04.884495 (+ 3416us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184}
W20260927 15:14:04.884965 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:04.876189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:54501 (local address 127.10.33.126:40655)
0927 15:14:04.876335 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:04.876340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:04.876411 (+ 71us) server_negotiation.cc:407] Connection header received
0927 15:14:04.876590 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:04.876596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:04.876674 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:04.876753 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:04.877414 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:04.878232 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:04.879571 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:04.880084 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:04.880197 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:04.884747 (+ 4550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.65:54501: BlockingRecv error: recv got EOF from 127.10.33.65:54501 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 15:14:04.885478 19355 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:14:04.924226 19492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:14:04.924743 19492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:14:04.924929 19492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:14:04.939674 19492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:14:04.939910 19492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:14:04.940126 19492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:14:04.959270 19492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/krb5kdc/oryx_127.10.33.67.keytab
--principal=oryx/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:46843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=41421
--tserver_master_addrs=127.10.33.126:40655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:14:04.961416 19492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:14:04.963524 19492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:14:04.980945 19499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:04.980985 19500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:04.981364 19502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:14:04.982741 19492 server_base.cc:1034] running on GCE node
I20260927 15:14:04.983419 19492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:14:04.984477 19492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:14:04.986066 19492 hybrid_clock.cc:648] HybridClock initialized: now 1790522044985916 us; error 118 us; skew 500 ppm
I20260927 15:14:04.986603 19492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:14:04 dist-test-slave-f5h6 krb5kdc[18620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522044, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:14:04.990190 19492 init.cc:377] Logged in from keytab as oryx/127.10.33.67@KRBTEST.COM (short username oryx)
I20260927 15:14:04.992537 19492 webserver.cc:460] Webserver started at http://127.10.33.67:41421/ using document root <none> and password file <none>
I20260927 15:14:04.993295 19492 fs_manager.cc:359] Metadata directory not provided
I20260927 15:14:04.993446 19492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:14:04.999631 19492 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:14:05.003108 19509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:05.004890 19492 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:14:05.005080 19492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "21c909b002c44d81871578de6c430be5"
format_stamp: "Formatted at 2026-09-27 15:14:02 on dist-test-slave-f5h6"
server_key: "067a06750d1cbf8df502e5058b9b9a07"
server_key_iv: "e383f4bfecd29b85c4fb9d52bddc0819"
server_key_version: "encryptionkey@0"
I20260927 15:14:05.006109 19492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:14:05.028728 19492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:14:05.033144 19492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:14:05.033402 19492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:14:05.034503 19492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:14:05.035974 19492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:14:05.036072 19492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:05.036180 19492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:14:05.036247 19492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:05.072736 19492 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:46843
I20260927 15:14:05.072789 19621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:46843 every 8 connection(s)
I20260927 15:14:05.074290 19492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:14:05.083727 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 19492
Sep 27 15:14:05 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:05 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:05.115762 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:05.084793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33642 (local address 127.10.33.126:40655)
0927 15:14:05.089794 (+ 5001us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:05.089801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:05.091032 (+ 1231us) server_negotiation.cc:407] Connection header received
0927 15:14:05.094174 (+ 3142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:05.094182 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:05.094311 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:05.094462 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:05.096642 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:05.098163 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:05.099957 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:05.100853 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:05.101028 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:05.105330 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:05.105345 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:05.105378 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:14:05.105465 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:05.110499 (+ 5034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:05.112162 (+ 1663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:05.112174 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:05.112186 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:05.112508 (+ 322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:05.112966 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:05.112972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:05.112976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:05.113768 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:05.114188 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:05.114756 (+ 568us) server_negotiation.cc:299] Negotiation successful
0927 15:14:05.115246 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4859,"thread_start_us":3274,"threads_started":1}
W20260927 15:14:05.513517 19490 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.110s user 0.007s sys 0.015s
W20260927 15:14:05.515100 19491 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:04.370007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:47021 (local address 127.10.33.126:40655)
0927 15:14:04.370416 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:04.370423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:04.379132 (+ 8709us) server_negotiation.cc:407] Connection header received
0927 15:14:04.391344 (+ 12212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:04.391351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:04.391579 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:04.391725 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:04.395399 (+ 3674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:04.397138 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:04.400715 (+ 3577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:04.401422 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:04.401576 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:05.514854 (+1113278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.66:47021: BlockingRecv error: recv got EOF from 127.10.33.66:47021 (error 108)
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":218,"threads_started":1}
I20260927 15:14:05.515996 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:04.369356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.66:47021)
0927 15:14:04.378913 (+ 9557us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:04.378986 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:04.390961 (+ 11975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:04.392561 (+ 1600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:04.392589 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:04.393628 (+ 1039us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:04.394833 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:04.394871 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:04.397426 (+ 2555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:04.397434 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:04.400432 (+ 2998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:04.400475 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:04.402143 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:04.402170 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:04.402615 (+ 445us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:04.403810 (+ 1195us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:04.403862 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:05.514618 (+1110756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":9155,"thread_start_us":10692,"threads_started":1}
W20260927 15:14:05.516913 19488 heartbeater.cc:644] Failed to heartbeat to 127.10.33.126:40655 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.33.126:40655: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Sep 27 15:14:05 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:05 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:05.529398 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:05.517553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.66:34773)
0927 15:14:05.517817 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:05.517841 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:05.518011 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:05.518360 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:05.518366 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:05.518906 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:05.519168 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:05.519181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:05.520536 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:05.520542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:05.522298 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:05.522320 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:05.523214 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:05.523221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:05.523423 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:05.524242 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:05.524268 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:05.529009 (+ 4741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98}
W20260927 15:14:05.529846 19491 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:05.517723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:34773 (local address 127.10.33.126:40655)
0927 15:14:05.517902 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:05.517909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:05.517931 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:14:05.518156 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:05.518160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:05.518224 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:05.518306 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:05.519357 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:05.520359 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:05.522520 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:05.523006 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:05.523132 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:05.529547 (+ 6415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.66:34773: BlockingRecv error: recv got EOF from 127.10.33.66:34773 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 15:14:05 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:05 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:05.541841 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:05.530589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.66:46011)
0927 15:14:05.530908 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:05.530931 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:05.531233 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:05.531711 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:05.531716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:05.532238 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:05.532573 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:05.532591 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:05.533697 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:05.533702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:05.535459 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:05.535482 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:05.536770 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:05.536783 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:05.536996 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:05.537825 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:05.537843 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:05.541572 (+ 3729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186}
W20260927 15:14:05.542089 19491 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:05.530846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:46011 (local address 127.10.33.126:40655)
0927 15:14:05.531008 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:05.531012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:05.531030 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:14:05.531470 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:05.531475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:05.531544 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:05.531633 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:05.532755 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:05.533551 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:05.535732 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:05.536318 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:05.536474 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:05.541809 (+ 5335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.66:46011: BlockingRecv error: recv got EOF from 127.10.33.66:46011 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 15:14:05.542248 19488 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 15:14:05 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:05 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:05.895839 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:05.886301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.65:60133)
0927 15:14:05.886832 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:05.886849 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:05.887022 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:05.887362 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:05.887367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:05.887768 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:05.887911 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:05.887921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:05.888891 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:05.888897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:05.889846 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:05.889886 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:05.890581 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:05.890588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:05.890767 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:05.891355 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:05.891373 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:05.895571 (+ 4198us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":320,"threads_started":1}
W20260927 15:14:05.896148 19491 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:05.886392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:60133 (local address 127.10.33.126:40655)
0927 15:14:05.886630 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:05.886637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:05.886934 (+ 297us) server_negotiation.cc:407] Connection header received
0927 15:14:05.887140 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:05.887145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:05.887227 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:05.887311 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:05.888057 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:05.888760 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:05.890012 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:05.890399 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:05.890519 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:05.895866 (+ 5347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.65:60133: BlockingRecv error: recv got EOF from 127.10.33.65:60133 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260927 15:14:06.100308 19624 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.012s user 0.005s sys 0.009s
W20260927 15:14:06.101411 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:05.077033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:55821 (local address 127.10.33.126:40655)
0927 15:14:05.077240 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:05.077246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:05.078344 (+ 1098us) server_negotiation.cc:407] Connection header received
0927 15:14:05.080211 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:05.080218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:05.080314 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:05.080479 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:05.082004 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:05.083210 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:05.085688 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:05.086464 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:05.086575 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:06.101201 (+1014626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.67:55821: BlockingRecv error: recv got EOF from 127.10.33.67:55821 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:14:06.101941 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:05.077357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.67:55821)
0927 15:14:05.078174 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:05.078220 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:05.079952 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:05.080723 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:05.080731 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:05.081234 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:05.081789 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:05.081807 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:05.083412 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:05.083418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:05.085461 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:05.085483 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:05.087000 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:05.087009 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:05.087376 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:05.088209 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:05.088242 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:06.101002 (+1012760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":313,"threads_started":1}
W20260927 15:14:06.102959 19622 heartbeater.cc:644] Failed to heartbeat to 127.10.33.126:40655 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.33.126:40655: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Sep 27 15:14:06 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:06 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:06.115345 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:06.103708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.67:42379)
0927 15:14:06.103901 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:06.103922 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:06.104040 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:06.104383 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:06.104392 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:06.104790 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:06.104956 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:06.104965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:06.106322 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:06.106326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:06.107341 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:06.107355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:06.108241 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:06.108248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:06.108383 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:06.109394 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:06.109422 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:06.115095 (+ 5673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 15:14:06.116294 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:06.103814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:42379 (local address 127.10.33.126:40655)
0927 15:14:06.104054 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:06.104059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:06.104102 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:14:06.104152 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:06.104156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:06.104213 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:06.104286 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:06.105148 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:06.105958 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:06.107562 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:06.108000 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:06.108188 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:06.116013 (+ 7825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.67:42379: BlockingRecv error: recv got EOF from 127.10.33.67:42379 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
Sep 27 15:14:06 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:06 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:06.126272 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:06.116352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.67:60069)
0927 15:14:06.116515 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:06.116933 (+ 418us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:06.117292 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:06.118238 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:06.118242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:06.118585 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:06.118759 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:06.118769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:06.119921 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:06.119925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:06.120894 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:06.120907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:06.121678 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:06.121683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:06.121818 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:06.122494 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:06.122514 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:06.126012 (+ 3498us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 15:14:06.126631 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:06.116971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:60069 (local address 127.10.33.126:40655)
0927 15:14:06.117499 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:06.117591 (+ 92us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:06.117683 (+ 92us) server_negotiation.cc:407] Connection header received
0927 15:14:06.117792 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:06.117798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:06.117941 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:06.118196 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:06.118955 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:06.119772 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:06.121049 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:06.121528 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:06.121635 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:06.126355 (+ 4720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.67:60069: BlockingRecv error: recv got EOF from 127.10.33.67:60069 (error 108)
Metrics: {"server-negotiator.queue_time_us":376}
W20260927 15:14:06.126753 19622 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 15:14:06 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:06 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:06.552345 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:06.543049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.66:37687)
0927 15:14:06.543585 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:06.543601 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:06.543723 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:06.544189 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:06.544193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:06.544642 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:06.544789 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:06.544798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:06.545822 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:06.545826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:06.546825 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:06.546837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:06.547510 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:06.547515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:06.547680 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:06.548287 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:06.548307 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:06.552043 (+ 3736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":305,"threads_started":1}
W20260927 15:14:06.552588 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:06.543153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:37687 (local address 127.10.33.126:40655)
0927 15:14:06.543445 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:06.543450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:06.543700 (+ 250us) server_negotiation.cc:407] Connection header received
0927 15:14:06.543857 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:06.543862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:06.543964 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:06.544105 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:06.544925 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:06.545683 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:06.546951 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:06.547329 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:06.547418 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:06.552312 (+ 4894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.66:37687: BlockingRecv error: recv got EOF from 127.10.33.66:37687 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 15:14:06 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:06 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:06.906733 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:06.897217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.65:54287)
0927 15:14:06.897703 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:06.897718 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:06.897892 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:06.898215 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:06.898220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:06.898652 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:06.898825 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:06.898835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:06.899866 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:06.899871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:06.900803 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:06.900815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:06.901515 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:06.901521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:06.901686 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:06.902447 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:06.902475 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:06.906407 (+ 3932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":260,"threads_started":1}
W20260927 15:14:06.906833 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:06.897360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:54287 (local address 127.10.33.126:40655)
0927 15:14:06.897624 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:06.897631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:06.897795 (+ 164us) server_negotiation.cc:407] Connection header received
0927 15:14:06.898021 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:06.898026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:06.898091 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:06.898172 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:06.898961 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:06.899737 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:06.900941 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:06.901365 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:06.901486 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:06.906606 (+ 5120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.65:54287: BlockingRecv error: recv got EOF from 127.10.33.65:54287 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 15:14:07 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:07 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:07.137399 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:07.127496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.67:44947)
0927 15:14:07.127997 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:07.128014 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:07.128167 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:07.128620 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:07.128623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:07.129024 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:07.129219 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:07.129231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:07.130807 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:07.130811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:07.132074 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:07.132091 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:07.132894 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:07.132899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:07.133040 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:07.133661 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:07.133680 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:07.137159 (+ 3479us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":295,"threads_started":1}
W20260927 15:14:07.137712 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:07.127700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:44947 (local address 127.10.33.126:40655)
0927 15:14:07.127935 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:07.127941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:07.128158 (+ 217us) server_negotiation.cc:407] Connection header received
0927 15:14:07.128328 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:07.128333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:07.128399 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:07.128496 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:07.129428 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:07.130632 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:07.132248 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:07.132689 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:07.132795 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:07.137398 (+ 4603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.67:44947: BlockingRecv error: recv got EOF from 127.10.33.67:44947 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 27 15:14:07 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:07 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:07.562556 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:07.553299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.66:59937)
0927 15:14:07.553776 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:07.553792 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:07.553942 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:07.554238 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:07.554241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:07.554723 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:07.554954 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:07.554966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:07.556076 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:07.556082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:07.557089 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:07.557101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:07.557978 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:07.557986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:07.558177 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:07.558856 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:07.558874 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:07.562300 (+ 3426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":286,"threads_started":1}
W20260927 15:14:07.562700 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:07.553456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:59937 (local address 127.10.33.126:40655)
0927 15:14:07.553714 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:07.553720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:07.553881 (+ 161us) server_negotiation.cc:407] Connection header received
0927 15:14:07.554062 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:07.554066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:07.554126 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:07.554208 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:07.555102 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:07.555910 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:07.557235 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:07.557734 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:07.557891 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:07.562519 (+ 4628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.66:59937: BlockingRecv error: recv got EOF from 127.10.33.66:59937 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 27 15:14:07 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:07 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:07.917457 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:07.907774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.65:42929)
0927 15:14:07.908313 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:07.908330 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:07.908517 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:07.908855 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:07.908860 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:07.909297 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:07.909481 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:07.909491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:07.910671 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:07.910677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:07.911670 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:07.911685 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:07.912452 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:07.912459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:07.912645 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:07.913256 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:07.913276 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:07.917097 (+ 3821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":263,"threads_started":1}
W20260927 15:14:07.917572 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:07.907977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:42929 (local address 127.10.33.126:40655)
0927 15:14:07.908184 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:07.908191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:07.908412 (+ 221us) server_negotiation.cc:407] Connection header received
0927 15:14:07.908646 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:07.908652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:07.908717 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:07.908808 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:07.909632 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:07.910506 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:07.911833 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:07.912291 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:07.912415 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:07.917331 (+ 4916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.65:42929: BlockingRecv error: recv got EOF from 127.10.33.65:42929 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 15:14:08 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:08 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:08.147614 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:08.138362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.67:53401)
0927 15:14:08.138842 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:08.138863 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:08.138973 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:08.139277 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:08.139280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:08.139668 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:08.139854 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:08.139863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:08.140873 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:08.140877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:08.141956 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:08.141971 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:08.142646 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:08.142651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:08.142785 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:08.143489 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:08.143509 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:08.147248 (+ 3739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":276,"threads_started":1}
W20260927 15:14:08.147680 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:08.138489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:53401 (local address 127.10.33.126:40655)
0927 15:14:08.138661 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:08.138665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:08.138960 (+ 295us) server_negotiation.cc:407] Connection header received
0927 15:14:08.139086 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:08.139090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:08.139164 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:08.139249 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:08.139971 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:08.140690 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:08.142092 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:08.142487 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:08.142585 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:08.147450 (+ 4865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.67:53401: BlockingRecv error: recv got EOF from 127.10.33.67:53401 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 15:14:08 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:08 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:08.572372 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:08.563416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.66:53633)
0927 15:14:08.563912 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:08.563926 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:08.564121 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:08.564421 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:08.564426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:08.564793 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:08.564954 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:08.564962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:08.565958 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:08.565972 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:08.566854 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:08.566865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:08.567558 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:08.567563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:08.567736 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:08.568339 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:08.568357 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:08.572138 (+ 3781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":284,"threads_started":1}
W20260927 15:14:08.572678 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:08.563512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:53633 (local address 127.10.33.126:40655)
0927 15:14:08.563711 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:08.563717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:08.564061 (+ 344us) server_negotiation.cc:407] Connection header received
0927 15:14:08.564232 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:08.564237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:08.564292 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:08.564368 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:08.565098 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:08.565824 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:08.567004 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:08.567425 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:08.567537 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:08.572439 (+ 4902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.66:53633: BlockingRecv error: recv got EOF from 127.10.33.66:53633 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 15:14:08 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:08 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:08.928056 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:08.918589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.65:48729)
0927 15:14:08.919235 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:08.919255 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:08.919421 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:08.919900 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:08.919905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:08.920342 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:08.920502 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:08.920512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:08.921692 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:08.921696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:08.922742 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:08.922756 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:08.923483 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:08.923490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:08.923678 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:08.924333 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:08.924353 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:08.927728 (+ 3375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":300,"threads_started":1}
W20260927 15:14:08.928148 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:08.918746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:48729 (local address 127.10.33.126:40655)
0927 15:14:08.918970 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:08.918976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:08.919358 (+ 382us) server_negotiation.cc:407] Connection header received
0927 15:14:08.919555 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:08.919561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:08.919719 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:08.919833 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:08.920643 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:08.921530 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:08.922886 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:08.923331 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:08.923445 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:08.927929 (+ 4484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.65:48729: BlockingRecv error: recv got EOF from 127.10.33.65:48729 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 15:14:09 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:09 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:09.157994 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:09.148732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.67:59741)
0927 15:14:09.149464 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:09.149479 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:09.149613 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:09.150025 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:09.150030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:09.150480 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:09.150710 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:09.150720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:09.151739 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:09.151752 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:09.152789 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:09.152802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:09.153485 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:09.153489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:09.153684 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:09.154292 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:09.154310 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:09.157713 (+ 3403us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":498,"threads_started":1}
W20260927 15:14:09.158245 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:09.148844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:59741 (local address 127.10.33.126:40655)
0927 15:14:09.149036 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:09.149043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:09.149619 (+ 576us) server_negotiation.cc:407] Connection header received
0927 15:14:09.149732 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:09.149737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:09.149817 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:09.149948 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:09.150875 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:09.151606 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:09.152938 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:09.153352 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:09.153451 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:09.157978 (+ 4527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.67:59741: BlockingRecv error: recv got EOF from 127.10.33.67:59741 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 15:14:09 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:09 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:09.588114 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:09.573576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.66:47407)
0927 15:14:09.574204 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:09.574219 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:09.574357 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:09.574723 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:09.574728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:09.575107 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:09.575268 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:09.575277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:09.576345 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:09.576350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:09.577342 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:09.577354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:09.578053 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:09.578059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:09.578226 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:09.584389 (+ 6163us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:09.584412 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:09.587848 (+ 3436us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":335,"threads_started":1}
W20260927 15:14:09.588368 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:09.573828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:47407 (local address 127.10.33.126:40655)
0927 15:14:09.574078 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:09.574085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:09.574312 (+ 227us) server_negotiation.cc:407] Connection header received
0927 15:14:09.574494 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:09.574498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:09.574565 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:09.574648 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:09.575412 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:09.576209 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:09.577492 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:09.577918 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:09.578029 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:09.588091 (+ 10062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.66:47407: BlockingRecv error: recv got EOF from 127.10.33.66:47407 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 27 15:14:09 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:09 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:09.938975 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:09.929168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.65:57455)
0927 15:14:09.929702 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:09.929718 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:09.929902 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:09.930247 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:09.930251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:09.930733 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:09.930918 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:09.930928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:09.932247 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:09.932253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:09.933244 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:09.933259 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:09.933983 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:09.933988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:09.934168 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:09.934799 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:09.934819 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:09.938644 (+ 3825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":265,"threads_started":1}
W20260927 15:14:09.939152 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:09.929380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:57455 (local address 127.10.33.126:40655)
0927 15:14:09.929638 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:09.929645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:09.929799 (+ 154us) server_negotiation.cc:407] Connection header received
0927 15:14:09.930026 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:09.930032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:09.930103 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:09.930191 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:09.931053 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:09.932094 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:09.933389 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:09.933821 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:09.933985 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:09.938886 (+ 4901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.65:57455: BlockingRecv error: recv got EOF from 127.10.33.65:57455 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 27 15:14:10 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:10 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:10.169364 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:10.159249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.67:49495)
0927 15:14:10.159753 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:10.159770 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:10.159892 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:10.160290 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:10.160295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:10.160743 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:10.160962 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:10.160976 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:10.162359 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:10.162363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:10.163383 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:10.163396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:10.164348 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:10.164353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:10.164498 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:10.165363 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:10.165389 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:10.169108 (+ 3719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":264,"threads_started":1}
W20260927 15:14:10.169721 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:10.159432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:49495 (local address 127.10.33.126:40655)
0927 15:14:10.159640 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:10.159645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:10.159896 (+ 251us) server_negotiation.cc:407] Connection header received
0927 15:14:10.160000 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:10.160005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:10.160084 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:10.160210 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:10.161164 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:10.162217 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:10.163570 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:10.164118 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:10.164306 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:10.169471 (+ 5165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.67:49495: BlockingRecv error: recv got EOF from 127.10.33.67:49495 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 15:14:10 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:10 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:10.598084 19641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:10.589072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.66:60983)
0927 15:14:10.589580 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:10.589597 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:10.589720 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:10.590072 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:10.590076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:10.590486 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:10.590630 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:10.590639 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:10.591762 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:10.591766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:10.592699 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:10.592712 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:10.593438 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:10.593442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:10.593576 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:10.594268 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:10.594288 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:10.597725 (+ 3437us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":279,"threads_started":1}
W20260927 15:14:10.598261 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:10.589312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:60983 (local address 127.10.33.126:40655)
0927 15:14:10.589543 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:10.589549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:10.589673 (+ 124us) server_negotiation.cc:407] Connection header received
0927 15:14:10.589805 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:10.589809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:10.589913 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:10.589996 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:10.590838 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:10.591596 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:10.592844 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:10.593294 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:10.593393 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:10.598002 (+ 4609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.66:60983: BlockingRecv error: recv got EOF from 127.10.33.66:60983 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 15:14:10 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:10 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:10.949172 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:10.939979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.65:51485)
0927 15:14:10.940431 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:10.940447 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:10.940557 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:10.940875 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:10.940879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:10.941329 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:10.941551 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:10.941562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:10.942646 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:10.942657 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:10.943594 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:10.943606 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:10.944335 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:10.944340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:10.944467 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:10.945079 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:10.945098 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:10.948871 (+ 3773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":246,"threads_started":1}
W20260927 15:14:10.949458 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:10.940061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:51485 (local address 127.10.33.126:40655)
0927 15:14:10.940314 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:10.940321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:10.940519 (+ 198us) server_negotiation.cc:407] Connection header received
0927 15:14:10.940649 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:10.940653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:10.940713 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:10.940792 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:10.941718 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:10.942472 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:10.943760 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:10.944200 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:10.944295 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:10.949224 (+ 4929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.65:51485: BlockingRecv error: recv got EOF from 127.10.33.65:51485 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 27 15:14:11 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:11 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:11.179299 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:11.170332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.67:37675)
0927 15:14:11.170884 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:11.170900 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:11.171023 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:11.171358 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:11.171361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:11.171810 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:11.171957 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:11.171965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:11.173017 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:11.173022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:11.174014 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:11.174028 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:11.174728 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:11.174735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:11.174881 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:11.175494 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:11.175512 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:11.179054 (+ 3542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":324,"threads_started":1}
W20260927 15:14:11.179562 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:11.170455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:37675 (local address 127.10.33.126:40655)
0927 15:14:11.170693 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:11.170697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:11.170999 (+ 302us) server_negotiation.cc:407] Connection header received
0927 15:14:11.171155 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:11.171159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:11.171225 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:11.171307 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:11.172083 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:11.172866 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:11.174176 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:11.174561 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:11.174671 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:11.179312 (+ 4641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.67:37675: BlockingRecv error: recv got EOF from 127.10.33.67:37675 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 15:14:11 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:11 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:11.609938 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:11.599579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.66:33717)
0927 15:14:11.600477 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:11.600505 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:11.600660 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:11.600976 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:11.600980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:11.601470 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:11.601631 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:11.601641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:11.602966 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:11.602971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:11.603947 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:11.603962 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:11.605045 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:11.605051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:11.605238 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:11.606255 (+ 1017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:11.606276 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:11.609603 (+ 3327us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":419,"threads_started":1}
W20260927 15:14:11.610153 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:11.599659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:33717 (local address 127.10.33.126:40655)
0927 15:14:11.599919 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:11.599926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:11.600662 (+ 736us) server_negotiation.cc:407] Connection header received
0927 15:14:11.600755 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:11.600760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:11.600838 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:11.600916 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:11.601803 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:11.602797 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:11.604111 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:11.604666 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:11.604844 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:11.609838 (+ 4994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.66:33717: BlockingRecv error: recv got EOF from 127.10.33.66:33717 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 15:14:11 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:11 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:11.959867 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:11.950285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.65:54987)
0927 15:14:11.950945 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:11.950965 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:11.951089 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:11.951407 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:11.951411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:11.951906 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:11.952083 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:11.952092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:11.953199 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:11.953203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:11.954316 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:11.954332 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:11.955182 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:11.955189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:11.955328 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:11.956073 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:11.956092 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:11.959569 (+ 3477us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":302,"threads_started":1}
W20260927 15:14:11.960018 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:11.950352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:54987 (local address 127.10.33.126:40655)
0927 15:14:11.950525 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:11.950531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:11.951081 (+ 550us) server_negotiation.cc:407] Connection header received
0927 15:14:11.951197 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:11.951201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:11.951267 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:11.951361 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:11.952232 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:11.953040 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:11.954527 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:11.954990 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:11.955089 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:11.959810 (+ 4721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.65:54987: BlockingRecv error: recv got EOF from 127.10.33.65:54987 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 15:14:12 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:12 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:12.190204 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:12.180306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.67:52095)
0927 15:14:12.180806 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:12.180825 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:12.180945 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:12.181380 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:12.181385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:12.181777 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:12.181991 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:12.182002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:12.183165 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:12.183170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:12.184116 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:12.184129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:12.184955 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:12.184960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:12.185112 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:12.185813 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:12.185832 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:12.189847 (+ 4015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":297,"threads_started":1}
W20260927 15:14:12.190313 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:12.180418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:52095 (local address 127.10.33.126:40655)
0927 15:14:12.180576 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:12.180581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:12.180891 (+ 310us) server_negotiation.cc:407] Connection header received
0927 15:14:12.181087 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:12.181091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:12.181176 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:12.181275 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:12.182197 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:12.182978 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:12.184309 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:12.184766 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:12.184886 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:12.190067 (+ 5181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.67:52095: BlockingRecv error: recv got EOF from 127.10.33.67:52095 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 15:14:12 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:12 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:12.621031 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:12.611224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.66:45559)
0927 15:14:12.611874 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:12.611890 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:12.612014 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:12.612410 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:12.612414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:12.612865 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:12.613028 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:12.613037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:12.614112 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:12.614118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:12.615067 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:12.615080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:12.616195 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:12.616202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:12.616352 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:12.617201 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:12.617226 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:12.620776 (+ 3550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":317,"threads_started":1}
W20260927 15:14:12.621306 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:12.611367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:45559 (local address 127.10.33.126:40655)
0927 15:14:12.611608 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:12.611615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:12.612015 (+ 400us) server_negotiation.cc:407] Connection header received
0927 15:14:12.612118 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:12.612122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:12.612192 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:12.612291 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:12.613202 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:12.613979 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:12.615229 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:12.616010 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:12.616136 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:12.620995 (+ 4859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.66:45559: BlockingRecv error: recv got EOF from 127.10.33.66:45559 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 27 15:14:12 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:12 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:12.970162 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:12.960872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.65:45201)
0927 15:14:12.961479 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:12.961498 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:12.961649 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:12.962116 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:12.962122 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:12.962536 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:12.962701 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:12.962711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:12.963750 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:12.963756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:12.964689 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:12.964702 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:12.965480 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:12.965487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:12.965653 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:12.966287 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:12.966305 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:12.969783 (+ 3478us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":261,"threads_started":1}
W20260927 15:14:12.970309 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:12.961077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:45201 (local address 127.10.33.126:40655)
0927 15:14:12.961387 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:12.961393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:12.961584 (+ 191us) server_negotiation.cc:407] Connection header received
0927 15:14:12.961845 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:12.961851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:12.961963 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:12.962066 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:12.962879 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:12.963593 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:12.964846 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:12.965303 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:12.965421 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:12.970059 (+ 4638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.65:45201: BlockingRecv error: recv got EOF from 127.10.33.65:45201 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 27 15:14:13 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:13 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:13.201660 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:13.191430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.67:50559)
0927 15:14:13.192053 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:13.192071 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:13.192208 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:13.192665 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:13.192670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:13.193155 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:13.193331 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:13.193342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:13.194665 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:13.194671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:13.196053 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:13.196068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:13.196942 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:13.196948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:13.197091 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:13.197656 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:13.197674 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:13.201355 (+ 3681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":262,"threads_started":1}
W20260927 15:14:13.202219 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:13.191695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:50559 (local address 127.10.33.126:40655)
0927 15:14:13.191907 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:13.191911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:13.192202 (+ 291us) server_negotiation.cc:407] Connection header received
0927 15:14:13.192378 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:13.192382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:13.192454 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:13.192545 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:13.193537 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:13.194492 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:13.196212 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:13.196793 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:13.196889 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:13.201728 (+ 4839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.67:50559: BlockingRecv error: recv got EOF from 127.10.33.67:50559 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 27 15:14:13 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:13 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:13.631390 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:13.622166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.66:59925)
0927 15:14:13.622678 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:13.622696 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:13.622809 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:13.623140 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:13.623144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:13.623520 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:13.623684 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:13.623693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:13.624886 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:13.624891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:13.625844 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:13.625895 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:13.626630 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:13.626636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:13.626756 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:13.627428 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:13.627447 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:13.631126 (+ 3679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":251,"threads_started":1}
W20260927 15:14:13.631722 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:13.622332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:59925 (local address 127.10.33.126:40655)
0927 15:14:13.622564 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:13.622569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:13.622794 (+ 225us) server_negotiation.cc:407] Connection header received
0927 15:14:13.622891 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:13.622895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:13.622959 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:13.623056 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:13.623832 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:13.624741 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:13.626030 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:13.626464 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:13.626561 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:13.631440 (+ 4879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.66:59925: BlockingRecv error: recv got EOF from 127.10.33.66:59925 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 15:14:13 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:13 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:13.979940 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:13.971136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.65:36063)
0927 15:14:13.971634 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:13.971652 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:13.971769 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:13.972063 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:13.972066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:13.972464 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:13.972600 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:13.972609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:13.973663 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:13.973668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:13.974663 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:13.974676 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:13.975434 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:13.975439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:13.975583 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:13.976250 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:13.976268 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:13.979665 (+ 3397us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":276,"threads_started":1}
W20260927 15:14:13.980347 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:13.971256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:36063 (local address 127.10.33.126:40655)
0927 15:14:13.971438 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:13.971443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:13.971753 (+ 310us) server_negotiation.cc:407] Connection header received
0927 15:14:13.971864 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:13.971868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:13.971950 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:13.972036 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:13.972759 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:13.973523 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:13.974809 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:13.975269 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:13.975372 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:13.979949 (+ 4577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.65:36063: BlockingRecv error: recv got EOF from 127.10.33.65:36063 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 15:14:14 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:14 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:14.213106 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:14.202735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.67:47697)
0927 15:14:14.203257 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:14.203272 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:14.203421 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:14.203730 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:14.203735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:14.204172 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:14.204379 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:14.204392 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:14.205698 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:14.205704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:14.207199 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:14.207219 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:14.208243 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:14.208250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:14.208448 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:14.209091 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:14.209119 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:14.212825 (+ 3706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":264,"threads_started":1}
W20260927 15:14:14.213291 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:14.202821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:47697 (local address 127.10.33.126:40655)
0927 15:14:14.203026 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:14.203033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:14.203367 (+ 334us) server_negotiation.cc:407] Connection header received
0927 15:14:14.203534 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:14.203538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:14.203600 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:14.203692 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:14.204516 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:14.205545 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:14.207387 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:14.208013 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:14.208180 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:14.213069 (+ 4889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.67:47697: BlockingRecv error: recv got EOF from 127.10.33.67:47697 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 15:14:14 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:14 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:14.645814 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:14.632619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.66:59587)
0927 15:14:14.633182 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:14.633198 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:14.633395 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:14.633757 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:14.633760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:14.634256 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:14.634452 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:14.634463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:14.635755 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:14.635761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:14.636945 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:14.636961 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:14.637793 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:14.637799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:14.638031 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:14.642059 (+ 4028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:14.642080 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:14.645569 (+ 3489us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":280,"threads_started":1}
W20260927 15:14:14.646188 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:14.632758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:59587 (local address 127.10.33.126:40655)
0927 15:14:14.633001 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:14.633007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:14.633373 (+ 366us) server_negotiation.cc:407] Connection header received
0927 15:14:14.633500 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:14.633504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:14.633568 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:14.633705 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:14.634602 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:14.635568 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:14.637110 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:14.637548 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:14.637653 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:14.645884 (+ 8231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.66:59587: BlockingRecv error: recv got EOF from 127.10.33.66:59587 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 15:14:14 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:14 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:14.990641 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:14.980963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.65:60483)
0927 15:14:14.981481 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:14.981499 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:14.981686 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:14.982054 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:14.982057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:14.982423 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:14.982577 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:14.982586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:14.983956 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:14.983962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:14.985217 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:14.985243 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:14.986046 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:14.986052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:14.986250 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:14.987006 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:14.987024 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:14.990405 (+ 3381us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":290,"threads_started":1}
W20260927 15:14:14.990818 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:14.981050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:60483 (local address 127.10.33.126:40655)
0927 15:14:14.981220 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:14.981225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:14.981666 (+ 441us) server_negotiation.cc:407] Connection header received
0927 15:14:14.981792 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:14.981797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:14.981879 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:14.981973 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:14.982744 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:14.983778 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:14.985391 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:14.985808 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:14.985997 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:14.990598 (+ 4601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.65:60483: BlockingRecv error: recv got EOF from 127.10.33.65:60483 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 15:14:15 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:15 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:15.223186 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:15.214087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.67:47985)
0927 15:14:15.214590 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:15.214606 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:15.214731 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:15.215071 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:15.215076 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:15.215471 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:15.215672 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:15.215682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:15.216803 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:15.216808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:15.217765 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:15.217777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:15.218521 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:15.218526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:15.218716 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:15.219309 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:15.219327 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:15.222865 (+ 3538us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":267,"threads_started":1}
W20260927 15:14:15.223316 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:15.214148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:47985 (local address 127.10.33.126:40655)
0927 15:14:15.214327 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:15.214331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:15.214766 (+ 435us) server_negotiation.cc:407] Connection header received
0927 15:14:15.214831 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:15.214834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:15.214903 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:15.214990 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:15.215852 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:15.216601 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:15.217941 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:15.218364 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:15.218459 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:15.223062 (+ 4603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.67:47985: BlockingRecv error: recv got EOF from 127.10.33.67:47985 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 15:14:15 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:15 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:15.656397 19656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:15.647005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.66:38467)
0927 15:14:15.647519 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:15.647536 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:15.647654 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:15.648066 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:15.648070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:15.648429 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:15.648589 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:15.648598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:15.649826 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:15.649831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:15.650968 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:15.650986 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:15.651833 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:15.651840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:15.652012 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:15.652633 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:15.652652 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:15.656065 (+ 3413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":283,"threads_started":1}
W20260927 15:14:15.656466 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:15.647192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:38467 (local address 127.10.33.126:40655)
0927 15:14:15.647366 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:15.647370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:15.647607 (+ 237us) server_negotiation.cc:407] Connection header received
0927 15:14:15.647815 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:15.647820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:15.647891 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:15.647984 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:15.648801 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:15.649641 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:15.651125 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:15.651672 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:15.651860 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:15.656258 (+ 4398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.66:38467: BlockingRecv error: recv got EOF from 127.10.33.66:38467 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 15:14:15 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:15 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:16.002085 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:15.991699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.65:42453)
0927 15:14:15.992287 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:15.992309 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:15.992468 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:15.992948 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:15.992953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:15.993628 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:15.993899 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:15.993914 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:15.995264 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:15.995270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:15.996330 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:15.996345 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:15.997125 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:15.997131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:15.997313 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:15.997956 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:15.997975 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:16.001808 (+ 3833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":529,"threads_started":1}
W20260927 15:14:16.002282 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:15.991929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:42453 (local address 127.10.33.126:40655)
0927 15:14:15.992216 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:15.992222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:15.992432 (+ 210us) server_negotiation.cc:407] Connection header received
0927 15:14:15.992680 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:15.992687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:15.992769 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:15.992879 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:15.994062 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:15.995111 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:15.996477 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:15.996968 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:15.997084 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:16.002062 (+ 4978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.65:42453: BlockingRecv error: recv got EOF from 127.10.33.65:42453 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 15:14:16 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:16 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:16.234134 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:16.224216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.67:57089)
0927 15:14:16.224881 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:16.224901 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:16.225053 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:16.225438 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:16.225441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:16.225789 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:16.225981 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:16.225991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:16.227421 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:16.227425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:16.228472 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:16.228486 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:16.229323 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:16.229328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:16.229448 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:16.230176 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:16.230194 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:16.233882 (+ 3688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":387,"threads_started":1}
W20260927 15:14:16.234378 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:16.224326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:57089 (local address 127.10.33.126:40655)
0927 15:14:16.224484 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:16.224491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:16.225000 (+ 509us) server_negotiation.cc:407] Connection header received
0927 15:14:16.225205 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:16.225211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:16.225288 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:16.225393 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:16.226232 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:16.227253 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:16.228634 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:16.229129 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:16.229239 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:16.234131 (+ 4892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.67:57089: BlockingRecv error: recv got EOF from 127.10.33.67:57089 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 15:14:16 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:16 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:16.666767 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:16.657495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.66:39319)
0927 15:14:16.658062 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:16.658080 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:16.658200 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:16.658547 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:16.658550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:16.658985 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:16.659161 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:16.659170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:16.660274 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:16.660278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:16.661379 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:16.661395 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:16.662305 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:16.662312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:16.662492 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:16.663084 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:16.663102 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:16.666490 (+ 3388us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":353,"threads_started":1}
W20260927 15:14:16.666925 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:16.657645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:39319 (local address 127.10.33.126:40655)
0927 15:14:16.657901 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:16.657907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:16.658186 (+ 279us) server_negotiation.cc:407] Connection header received
0927 15:14:16.658344 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:16.658348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:16.658411 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:16.658511 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:16.659337 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:16.660118 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:16.661545 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:16.662143 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:16.662269 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:16.666695 (+ 4426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.66:39319: BlockingRecv error: recv got EOF from 127.10.33.66:39319 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 15:14:17 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:17 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:17.012676 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:17.003139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.65:51071)
0927 15:14:17.003735 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:17.003751 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:17.003889 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:17.004219 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:17.004224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:17.004658 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:17.004832 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:17.004841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:17.005841 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:17.005845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:17.006806 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:17.006819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:17.007553 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:17.007559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:17.007727 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:17.008413 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:17.008431 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:17.012344 (+ 3913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":373,"threads_started":1}
W20260927 15:14:17.012744 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:17.003280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:51071 (local address 127.10.33.126:40655)
0927 15:14:17.003500 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:17.003507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:17.003836 (+ 329us) server_negotiation.cc:407] Connection header received
0927 15:14:17.004012 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:17.004018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:17.004085 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:17.004199 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:17.004969 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:17.005701 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:17.006939 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:17.007379 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:17.007490 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:17.012509 (+ 5019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.65:51071: BlockingRecv error: recv got EOF from 127.10.33.65:51071 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 15:14:17 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:17 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:17.244496 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:17.235292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.67:34311)
0927 15:14:17.235808 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:17.235823 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:17.235953 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:17.236320 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:17.236324 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:17.236790 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:17.236990 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:17.237002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:17.238065 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:17.238069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:17.238986 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:17.238997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:17.239714 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:17.239719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:17.239879 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:17.240588 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:17.240606 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:17.244147 (+ 3541us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":278,"threads_started":1}
W20260927 15:14:17.244681 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:17.235360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:34311 (local address 127.10.33.126:40655)
0927 15:14:17.235539 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:17.235546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:17.235906 (+ 360us) server_negotiation.cc:407] Connection header received
0927 15:14:17.236061 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:17.236064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:17.236160 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:17.236264 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:17.237131 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:17.237944 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:17.239125 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:17.239542 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:17.239644 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:17.244388 (+ 4744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.67:34311: BlockingRecv error: recv got EOF from 127.10.33.67:34311 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 15:14:17 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:17 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:17.677510 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:17.667984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.66:39057)
0927 15:14:17.668541 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:17.668556 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:17.668710 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:17.669074 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:17.669079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:17.669491 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:17.669655 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:17.669666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:17.670968 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:17.670974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:17.672005 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:17.672019 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:17.672978 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:17.672984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:17.673122 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:17.673751 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:17.673771 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:17.677267 (+ 3496us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":305,"threads_started":1}
W20260927 15:14:17.677687 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:17.668077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:39057 (local address 127.10.33.126:40655)
0927 15:14:17.668353 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:17.668359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:17.668648 (+ 289us) server_negotiation.cc:407] Connection header received
0927 15:14:17.668834 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:17.668838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:17.668922 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:17.669009 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:17.669847 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:17.670791 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:17.672135 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:17.672733 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:17.672834 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:17.677483 (+ 4649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.66:39057: BlockingRecv error: recv got EOF from 127.10.33.66:39057 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 27 15:14:18 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:18 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:18.023288 19663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:18.013779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.65:49927)
0927 15:14:18.014372 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:18.014392 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:18.014527 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:18.014846 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:18.014849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:18.015225 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:18.015368 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:18.015377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:18.016469 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:18.016474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:18.017442 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:18.017455 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:18.018259 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:18.018264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:18.018411 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:18.019148 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:18.019173 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:18.022968 (+ 3795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":370,"threads_started":1}
W20260927 15:14:18.023449 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:18.014093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:49927 (local address 127.10.33.126:40655)
0927 15:14:18.014297 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:18.014301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:18.014508 (+ 207us) server_negotiation.cc:407] Connection header received
0927 15:14:18.014633 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:18.014638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:18.014721 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:18.014815 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:18.015512 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:18.016321 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:18.017604 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:18.018093 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:18.018198 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:18.023230 (+ 5032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.65:49927: BlockingRecv error: recv got EOF from 127.10.33.65:49927 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 15:14:18 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:18 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:18.254680 19664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:18.245726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.67:59985)
0927 15:14:18.246237 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:18.246253 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:18.246375 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:18.246706 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:18.246709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:18.247094 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:18.247262 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:18.247271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:18.248332 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:18.248336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:18.249255 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:18.249266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:18.250050 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:18.250056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:18.250220 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:18.250790 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:18.250807 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:18.254386 (+ 3579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":281,"threads_started":1}
W20260927 15:14:18.254837 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:18.245927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:59985 (local address 127.10.33.126:40655)
0927 15:14:18.246138 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:18.246145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:18.246355 (+ 210us) server_negotiation.cc:407] Connection header received
0927 15:14:18.246465 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:18.246469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:18.246533 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:18.246614 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:18.247405 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:18.248199 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:18.249414 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:18.249824 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:18.249960 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:18.254616 (+ 4656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.67:59985: BlockingRecv error: recv got EOF from 127.10.33.67:59985 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 15:14:18 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:18 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:18.688081 19665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:18.678521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.66:33141)
0927 15:14:18.678989 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:18.679005 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:18.679123 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:18.679445 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:18.679449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:18.679919 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:18.680091 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:18.680100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:18.681153 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:18.681157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:18.682236 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:18.682249 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:18.683004 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:18.683009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:18.683146 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:18.683916 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:18.683936 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:18.687780 (+ 3844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":263,"threads_started":1}
W20260927 15:14:18.688246 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:18.678707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:33141 (local address 127.10.33.126:40655)
0927 15:14:18.678947 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:18.678953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:18.679074 (+ 121us) server_negotiation.cc:407] Connection header received
0927 15:14:18.679233 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:18.679236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:18.679297 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:18.679372 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:18.680250 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:18.681015 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:18.682389 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:18.682852 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:18.682958 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:18.687992 (+ 5034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.66:33141: BlockingRecv error: recv got EOF from 127.10.33.66:33141 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 27 15:14:19 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:19 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:19.033776 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:19.024493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.65:44861)
0927 15:14:19.024998 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:19.025014 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:19.025136 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:19.025452 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:19.025457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:19.025818 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:19.026039 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:19.026049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:19.027220 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:19.027225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:19.028135 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:19.028147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:19.028837 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:19.028843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:19.029012 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:19.029616 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:19.029640 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:19.033424 (+ 3784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":283,"threads_started":1}
W20260927 15:14:19.034008 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:19.024582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:44861 (local address 127.10.33.126:40655)
0927 15:14:19.024757 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:19.024763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:19.025101 (+ 338us) server_negotiation.cc:407] Connection header received
0927 15:14:19.025249 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:19.025255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:19.025323 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:19.025412 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:19.026180 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:19.027034 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:19.028278 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:19.028703 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:19.028807 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:19.033682 (+ 4875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.65:44861: BlockingRecv error: recv got EOF from 127.10.33.65:44861 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 15:14:19 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:19 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.67@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:19.264693 19667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:19.255738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.67:56013)
0927 15:14:19.256261 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:19.256277 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:19.256412 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:19.256753 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:19.256758 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:19.257170 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:19.257307 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:19.257316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:19.258394 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:19.258399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:19.259303 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:19.259314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:19.260002 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:19.260008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:19.260176 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:19.260862 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:19.260881 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:19.264464 (+ 3583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":292,"threads_started":1}
W20260927 15:14:19.264863 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:19.255847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:56013 (local address 127.10.33.126:40655)
0927 15:14:19.256123 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:19.256129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:19.256352 (+ 223us) server_negotiation.cc:407] Connection header received
0927 15:14:19.256529 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:19.256534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:19.256594 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:19.256704 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:19.257441 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:19.258256 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:19.259431 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:19.259865 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:19.259972 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:19.264654 (+ 4682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.67:56013: BlockingRecv error: recv got EOF from 127.10.33.67:56013 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
Sep 27 15:14:19 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:19 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.66@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:19.703485 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:19.689362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.66:47657)
0927 15:14:19.689905 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:19.689926 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:19.690062 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:19.690441 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:19.690445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:19.690820 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:19.691009 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:19.691019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:19.692104 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:19.692108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:19.693032 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:19.693045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:19.693775 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:19.693780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:19.693943 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:19.699558 (+ 5615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:19.699580 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:19.703187 (+ 3607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":273,"threads_started":1}
W20260927 15:14:19.703660 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:19.689533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:47657 (local address 127.10.33.126:40655)
0927 15:14:19.689764 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:19.689770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:19.690034 (+ 264us) server_negotiation.cc:407] Connection header received
0927 15:14:19.690170 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:19.690174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:19.690236 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:19.690349 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:19.691152 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:19.691953 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:19.693165 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:19.693606 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:19.693698 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:19.703464 (+ 9766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.66:47657: BlockingRecv error: recv got EOF from 127.10.33.66:47657 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 15:14:20 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:14:20 dist-test-slave-f5h6 krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.33.65@KRBTEST.COM for oryx/127.10.33.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:14:20.044782 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:20.034789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40655 (local address 127.10.33.65:47857)
0927 15:14:20.035384 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:20.035401 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:20.035518 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:20.035855 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:20.035860 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:20.036254 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:20.036446 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:20.036456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:20.037715 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:20.037719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:20.038738 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:20.038754 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:20.039666 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:20.039671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:20.039832 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:20.040485 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:20.040503 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:20.044449 (+ 3946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.33.126:40655: Server oryx/127.10.33.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":344,"threads_started":1}
W20260927 15:14:20.045014 19357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:20.034902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:47857 (local address 127.10.33.126:40655)
0927 15:14:20.035150 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:20.035158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:20.035478 (+ 320us) server_negotiation.cc:407] Connection header received
0927 15:14:20.035609 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:20.035612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:20.035674 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:20.035802 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:20.036608 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:20.037463 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:20.038911 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:20.039434 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:20.039640 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:20.044684 (+ 5044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.33.65:47857: BlockingRecv error: recv got EOF from 127.10.33.65:47857 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 15:14:20.093916 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 19208
I20260927 15:14:20.107693 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:20.095406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33752 (local address 127.10.33.65:37951)
0927 15:14:20.096089 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:20.096097 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:20.096128 (+ 31us) server_negotiation.cc:407] Connection header received
0927 15:14:20.096191 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:20.096197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:20.096479 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:20.096682 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:20.098317 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:20.099533 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:20.100880 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:20.101437 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:20.101571 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:20.104155 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:20.104165 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:20.104182 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:14:20.104228 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:20.106862 (+ 2634us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.33.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 15:14:20.107262 (+ 400us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33752: Request ticket server kudu/127.10.33.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":332,"threads_started":1}
W20260927 15:14:20.107821 19671 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33752: Request ticket server kudu/127.10.33.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 15:14:20.107823 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 19208: Not authorized: Client connection negotiation failed: client connection to 127.10.33.65:37951: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.33.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 15:14:20.108016 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 19208
I20260927 15:14:20.122159 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 19359
I20260927 15:14:20.135783 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:20.123387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47552 (local address 127.10.33.66:36151)
0927 15:14:20.124051 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:20.124058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:20.124088 (+ 30us) server_negotiation.cc:407] Connection header received
0927 15:14:20.124150 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:20.124157 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:20.124406 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:20.124571 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:20.126430 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:20.127493 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:20.128725 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:20.129332 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:20.129449 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:20.132023 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:20.132037 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:20.132060 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:14:20.132117 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:20.135181 (+ 3064us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.33.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 15:14:20.135529 (+ 348us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47552: Request ticket server kudu/127.10.33.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":349,"threads_started":1}
W20260927 15:14:20.135881 19672 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47552: Request ticket server kudu/127.10.33.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 15:14:20.136014 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 19359: Not authorized: Client connection negotiation failed: client connection to 127.10.33.66:36151: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.33.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 15:14:20.136164 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 19359
I20260927 15:14:20.149971 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 19492
I20260927 15:14:20.162624 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:20.151061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59212 (local address 127.10.33.67:46843)
0927 15:14:20.151687 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:20.151698 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:20.151729 (+ 31us) server_negotiation.cc:407] Connection header received
0927 15:14:20.151814 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:20.151823 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:20.152151 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:20.152351 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:20.153688 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:20.154790 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:20.156103 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:20.156646 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:20.156745 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:20.159260 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:20.159269 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:20.159284 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:14:20.159325 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:20.162059 (+ 2734us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.33.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 15:14:20.162377 (+ 318us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59212: Request ticket server kudu/127.10.33.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":301,"threads_started":1}
W20260927 15:14:20.162714 19673 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59212: Request ticket server kudu/127.10.33.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 15:14:20.163035 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 19492: Not authorized: Client connection negotiation failed: client connection to 127.10.33.67:46843: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.33.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 15:14:20.163177 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 19492
I20260927 15:14:20.177100 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 19139
I20260927 15:14:20.178521 19201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:14:20.296103 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 19139
2026-09-27T15:14:20Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21388 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T15:14:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:14:20Z Disabled control of system clock
I20260927 15:14:20.372650 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:42211
--webserver_interface=127.10.33.126
--webserver_port=0
--builtin_ntp_servers=127.10.33.84:39055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:42211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 15:14:20.713204 19678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:14:20.713541 19678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:14:20.713618 19678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:14:20.721108 19678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:14:20.721233 19678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:14:20.721275 19678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:14:20.721321 19678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:14:20.721364 19678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:14:20.731038 19678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:42211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:42211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:14:20.732304 19678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:14:20.734058 19678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:14:20.745543 19686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:20.745671 19683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:20.745983 19684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:14:20.747264 19678 server_base.cc:1034] running on GCE node
I20260927 15:14:20.748183 19678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:14:20.749554 19678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:14:20.750742 19678 hybrid_clock.cc:648] HybridClock initialized: now 1790522060750698 us; error 38 us; skew 500 ppm
I20260927 15:14:20.751103 19678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:14:20.753355 19678 webserver.cc:460] Webserver started at http://127.10.33.126:35345/ using document root <none> and password file <none>
I20260927 15:14:20.754077 19678 fs_manager.cc:359] Metadata directory not provided
I20260927 15:14:20.754179 19678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:14:20.754479 19678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:14:20.757341 19678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "15b58dcfd13d4b7b8807943a29ca016a"
format_stamp: "Formatted at 2026-09-27 15:14:20 on dist-test-slave-f5h6"
server_key: "2b34c39abcbeb4da6091d3a46c3dc606"
server_key_iv: "bf3227cc58a2badc39501edc1a769e80"
server_key_version: "encryptionkey@0"
I20260927 15:14:20.758167 19678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "15b58dcfd13d4b7b8807943a29ca016a"
format_stamp: "Formatted at 2026-09-27 15:14:20 on dist-test-slave-f5h6"
server_key: "2b34c39abcbeb4da6091d3a46c3dc606"
server_key_iv: "bf3227cc58a2badc39501edc1a769e80"
server_key_version: "encryptionkey@0"
I20260927 15:14:20.763346 19678 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 15:14:20.767515 19692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:20.769115 19678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:14:20.769309 19678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "15b58dcfd13d4b7b8807943a29ca016a"
format_stamp: "Formatted at 2026-09-27 15:14:20 on dist-test-slave-f5h6"
server_key: "2b34c39abcbeb4da6091d3a46c3dc606"
server_key_iv: "bf3227cc58a2badc39501edc1a769e80"
server_key_version: "encryptionkey@0"
I20260927 15:14:20.769543 19678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:14:20.817471 19678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:14:20.818493 19678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:14:20.818820 19678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:14:20.837893 19678 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:42211
I20260927 15:14:20.837916 19743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:42211 every 8 connection(s)
I20260927 15:14:20.839258 19678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:14:20.843317 19744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:14:20.848171 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 19678
I20260927 15:14:20.848378 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:14:20.849139 10373 external_mini_cluster.cc:1428] Setting key 011ee9b096949ef04abbf98e4617ec2c
I20260927 15:14:20.855525 19744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a: Bootstrap starting.
I20260927 15:14:20.859750 19744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a: Neither blocks nor log segments found. Creating new log.
I20260927 15:14:20.861222 19744 log.cc:824] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:14:20.865085 19744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a: No bootstrap required, opened a new log
I20260927 15:14:20.865974 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:20.851336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.10.33.126:42211)
0927 15:14:20.852491 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:20.852510 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:20.852561 (+ 51us) server_negotiation.cc:407] Connection header received
0927 15:14:20.854945 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:20.854976 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:20.855643 (+ 667us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:20.856147 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:20.856693 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:20.858632 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:20.860327 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:20.861424 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:20.861617 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:20.862962 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:20.862973 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:20.862994 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:14:20.863705 (+ 711us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:20.863808 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:14:20.863839 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:20.863959 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:20.864190 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 15:14:20.864536 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":383,"threads_started":1}
I20260927 15:14:20.872741 19744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b58dcfd13d4b7b8807943a29ca016a" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42211 } }
I20260927 15:14:20.873096 19744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:14:20.873211 19744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15b58dcfd13d4b7b8807943a29ca016a, State: Initialized, Role: FOLLOWER
I20260927 15:14:20.873826 19744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b58dcfd13d4b7b8807943a29ca016a" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42211 } }
I20260927 15:14:20.874085 19744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:14:20.874270 19744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:14:20.874434 19744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:14:20.876565 19744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b58dcfd13d4b7b8807943a29ca016a" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42211 } }
I20260927 15:14:20.877027 19744 leader_election.cc:302] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15b58dcfd13d4b7b8807943a29ca016a; no voters:
I20260927 15:14:20.877574 19744 leader_election.cc:288] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:14:20.877701 19749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:14:20.878369 19749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a [term 1 LEADER]: Becoming Leader. State: Replica: 15b58dcfd13d4b7b8807943a29ca016a, State: Running, Role: LEADER
I20260927 15:14:20.878901 19749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b58dcfd13d4b7b8807943a29ca016a" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42211 } }
I20260927 15:14:20.879606 19744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:14:20.881927 19751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15b58dcfd13d4b7b8807943a29ca016a. Latest consensus state: current_term: 1 leader_uuid: "15b58dcfd13d4b7b8807943a29ca016a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b58dcfd13d4b7b8807943a29ca016a" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42211 } } }
I20260927 15:14:20.882438 19751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a [sys.catalog]: This master's current role is: LEADER
I20260927 15:14:20.882501 19750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15b58dcfd13d4b7b8807943a29ca016a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b58dcfd13d4b7b8807943a29ca016a" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42211 } } }
I20260927 15:14:20.882817 19750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a [sys.catalog]: This master's current role is: LEADER
I20260927 15:14:20.884318 19756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:14:20.889726 19756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:14:20.899776 19756 catalog_manager.cc:1347] Generated new cluster ID: 96960eb6bbbd4bd89260ef32d7233d7f
I20260927 15:14:20.899905 19756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:14:20.940994 19756 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:14:20.943229 19756 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:14:20.955792 19756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15b58dcfd13d4b7b8807943a29ca016a: Generated new TSK 0
I20260927 15:14:20.957293 19756 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:14:20.969735 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:42211
--builtin_ntp_servers=127.10.33.84:39055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 15:14:21.367578 19768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:14:21.367884 19768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:14:21.367998 19768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:14:21.375787 19768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:14:21.375916 19768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:14:21.376026 19768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:14:21.386312 19768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:42211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:14:21.387672 19768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:14:21.390317 19768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:14:21.405113 19774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:21.405171 19776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:21.405277 19773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:14:21.406001 19768 server_base.cc:1034] running on GCE node
I20260927 15:14:21.406644 19768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:14:21.407801 19768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:14:21.409049 19768 hybrid_clock.cc:648] HybridClock initialized: now 1790522061409000 us; error 45 us; skew 500 ppm
I20260927 15:14:21.409407 19768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:14:21.411996 19768 webserver.cc:460] Webserver started at http://127.10.33.65:34767/ using document root <none> and password file <none>
I20260927 15:14:21.412750 19768 fs_manager.cc:359] Metadata directory not provided
I20260927 15:14:21.412858 19768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:14:21.413183 19768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:14:21.416343 19768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "b4c8221ee4164e4791f450e43861f27a"
format_stamp: "Formatted at 2026-09-27 15:14:21 on dist-test-slave-f5h6"
server_key: "840db77d5f29f57adfa54870b6217679"
server_key_iv: "05cc5be341e75b29e92a1b6ba11377f9"
server_key_version: "encryptionkey@0"
I20260927 15:14:21.417235 19768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "b4c8221ee4164e4791f450e43861f27a"
format_stamp: "Formatted at 2026-09-27 15:14:21 on dist-test-slave-f5h6"
server_key: "840db77d5f29f57adfa54870b6217679"
server_key_iv: "05cc5be341e75b29e92a1b6ba11377f9"
server_key_version: "encryptionkey@0"
I20260927 15:14:21.423774 19768 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260927 15:14:21.429381 19782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:21.431172 19768 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:14:21.431387 19768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "b4c8221ee4164e4791f450e43861f27a"
format_stamp: "Formatted at 2026-09-27 15:14:21 on dist-test-slave-f5h6"
server_key: "840db77d5f29f57adfa54870b6217679"
server_key_iv: "05cc5be341e75b29e92a1b6ba11377f9"
server_key_version: "encryptionkey@0"
I20260927 15:14:21.431627 19768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:14:21.481194 19768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:14:21.482118 19768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:14:21.482414 19768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:14:21.483552 19768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:14:21.485313 19768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:14:21.485440 19768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:21.485563 19768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:14:21.485625 19768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:21.519402 19768 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:34315
I20260927 15:14:21.519434 19894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:34315 every 8 connection(s)
I20260927 15:14:21.520651 19768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:14:21.521199 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 19768
I20260927 15:14:21.521394 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:14:21.522192 10373 external_mini_cluster.cc:1428] Setting key ae279d577503df50f58f625a9c0b5c53
I20260927 15:14:21.525283 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:42211
--builtin_ntp_servers=127.10.33.84:39055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 15:14:21.535200 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:21.523300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:36979 (local address 127.10.33.126:42211)
0927 15:14:21.523771 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:21.523777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:21.525284 (+ 1507us) server_negotiation.cc:407] Connection header received
0927 15:14:21.527764 (+ 2480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:21.527769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:21.527847 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:21.527945 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:21.529311 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:21.530084 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:21.532304 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:21.532718 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:21.532858 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:21.534456 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:21.534461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:21.534465 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:14:21.534486 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:21.534525 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:14:21.534539 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:21.534621 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:21.534859 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 15:14:21.534943 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":218,"threads_started":1}
I20260927 15:14:21.536811 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:21.523903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:42211 (local address 127.10.33.65:36979)
0927 15:14:21.525036 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:21.525100 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:21.527405 (+ 2305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:21.528210 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:21.528222 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:21.528340 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:21.529103 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:21.529126 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:21.530223 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:21.530228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:21.532073 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:21.532096 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:21.533006 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:21.533014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:21.533240 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:21.534018 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:14:21.534022 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:21.534092 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:14:21.534199 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:14:21.534637 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:14:21.534647 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:14:21.534656 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 15:14:21.534853 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 15:14:21.535296 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":281,"threads_started":1}
I20260927 15:14:21.538506 19895 heartbeater.cc:347] Connected to a master server at 127.10.33.126:42211
I20260927 15:14:21.538813 19895 heartbeater.cc:464] Registering TS with master...
I20260927 15:14:21.539808 19895 heartbeater.cc:511] Master 127.10.33.126:42211 requested a full tablet report, sending...
I20260927 15:14:21.542353 19709 ts_manager.cc:194] Registered new tserver with Master: b4c8221ee4164e4791f450e43861f27a (127.10.33.65:34315)
I20260927 15:14:21.543890 19709 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.33.65:36979
W20260927 15:14:21.859014 19899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:14:21.859319 19899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:14:21.859426 19899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:14:21.866659 19899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:14:21.866770 19899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:14:21.866921 19899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:14:21.875985 19899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:42211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:14:21.877225 19899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:14:21.878932 19899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:14:21.891904 19904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:21.891904 19905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:21.891953 19907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:14:21.892850 19899 server_base.cc:1034] running on GCE node
I20260927 15:14:21.893472 19899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:14:21.894440 19899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:14:21.895656 19899 hybrid_clock.cc:648] HybridClock initialized: now 1790522061895611 us; error 30 us; skew 500 ppm
I20260927 15:14:21.896003 19899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:14:21.898253 19899 webserver.cc:460] Webserver started at http://127.10.33.66:34233/ using document root <none> and password file <none>
I20260927 15:14:21.898972 19899 fs_manager.cc:359] Metadata directory not provided
I20260927 15:14:21.899058 19899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:14:21.899348 19899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:14:21.902241 19899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "8df492c3db034a59837b5ef754453f60"
format_stamp: "Formatted at 2026-09-27 15:14:21 on dist-test-slave-f5h6"
server_key: "ca7a9fc761c07e75a0f0613cdbbc0b90"
server_key_iv: "2580c3c14ab7bc69d7f60eccdb6912b2"
server_key_version: "encryptionkey@0"
I20260927 15:14:21.903005 19899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "8df492c3db034a59837b5ef754453f60"
format_stamp: "Formatted at 2026-09-27 15:14:21 on dist-test-slave-f5h6"
server_key: "ca7a9fc761c07e75a0f0613cdbbc0b90"
server_key_iv: "2580c3c14ab7bc69d7f60eccdb6912b2"
server_key_version: "encryptionkey@0"
I20260927 15:14:21.908216 19899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:14:21.911875 19913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:21.913174 19899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:14:21.913327 19899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "8df492c3db034a59837b5ef754453f60"
format_stamp: "Formatted at 2026-09-27 15:14:21 on dist-test-slave-f5h6"
server_key: "ca7a9fc761c07e75a0f0613cdbbc0b90"
server_key_iv: "2580c3c14ab7bc69d7f60eccdb6912b2"
server_key_version: "encryptionkey@0"
I20260927 15:14:21.913522 19899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:14:21.926389 19899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:14:21.927203 19899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:14:21.927492 19899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:14:21.928471 19899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:14:21.929925 19899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:14:21.930017 19899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:21.930123 19899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:14:21.930190 19899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:21.959609 19899 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:43763
I20260927 15:14:21.959652 20025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:43763 every 8 connection(s)
I20260927 15:14:21.960873 19899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:14:21.961473 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 19899
I20260927 15:14:21.961643 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:14:21.962235 10373 external_mini_cluster.cc:1428] Setting key e050b5ed4bea545f8ada4b16f19621ba
I20260927 15:14:21.965281 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:42211
--builtin_ntp_servers=127.10.33.84:39055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 15:14:21.975410 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:21.963514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:37477 (local address 127.10.33.126:42211)
0927 15:14:21.963699 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:21.963709 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:21.965343 (+ 1634us) server_negotiation.cc:407] Connection header received
0927 15:14:21.967849 (+ 2506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:21.967853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:21.967927 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:21.968016 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:21.969460 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:21.970271 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:21.972462 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:21.972865 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:21.973000 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:21.974568 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:21.974573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:21.974577 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:14:21.974599 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:21.974631 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:14:21.974643 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:21.974717 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:21.974982 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 15:14:21.975084 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:14:21.976500 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:21.964102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:42211 (local address 127.10.33.66:37477)
0927 15:14:21.965168 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:21.965214 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:21.967500 (+ 2286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:21.968269 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:21.968323 (+ 54us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:21.968459 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:21.969235 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:21.969258 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:21.970427 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:21.970432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:21.972283 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:21.972300 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:21.973159 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:21.973168 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:21.973414 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:21.974208 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:14:21.974210 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:21.974269 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:14:21.974379 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:14:21.974728 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:14:21.974743 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:14:21.974752 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 15:14:21.974968 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 15:14:21.975402 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":282,"threads_started":1}
I20260927 15:14:21.978048 20026 heartbeater.cc:347] Connected to a master server at 127.10.33.126:42211
I20260927 15:14:21.978348 20026 heartbeater.cc:464] Registering TS with master...
I20260927 15:14:21.979106 20026 heartbeater.cc:511] Master 127.10.33.126:42211 requested a full tablet report, sending...
I20260927 15:14:21.980715 19709 ts_manager.cc:194] Registered new tserver with Master: 8df492c3db034a59837b5ef754453f60 (127.10.33.66:43763)
I20260927 15:14:21.981797 19709 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.33.66:37477
W20260927 15:14:22.291198 20029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:14:22.291498 20029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:14:22.291579 20029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:14:22.299129 20029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:14:22.299269 20029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:14:22.299439 20029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:14:22.308784 20029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:42211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:14:22.310222 20029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:14:22.312041 20029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:14:22.325740 20034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:22.325639 20037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:22.325956 20035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:14:22.326474 20029 server_base.cc:1034] running on GCE node
I20260927 15:14:22.327102 20029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:14:22.328116 20029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:14:22.329361 20029 hybrid_clock.cc:648] HybridClock initialized: now 1790522062329300 us; error 53 us; skew 500 ppm
I20260927 15:14:22.329687 20029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:14:22.331951 20029 webserver.cc:460] Webserver started at http://127.10.33.67:37237/ using document root <none> and password file <none>
I20260927 15:14:22.332643 20029 fs_manager.cc:359] Metadata directory not provided
I20260927 15:14:22.332741 20029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:14:22.333050 20029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:14:22.336404 20029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "0ab78f4caf814f26a025d6ef15d43d32"
format_stamp: "Formatted at 2026-09-27 15:14:22 on dist-test-slave-f5h6"
server_key: "a63b2956bcab4ef2bba0cb449b175b47"
server_key_iv: "a986bd9517cd807d14b7c66b33f769fd"
server_key_version: "encryptionkey@0"
I20260927 15:14:22.337292 20029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "0ab78f4caf814f26a025d6ef15d43d32"
format_stamp: "Formatted at 2026-09-27 15:14:22 on dist-test-slave-f5h6"
server_key: "a63b2956bcab4ef2bba0cb449b175b47"
server_key_iv: "a986bd9517cd807d14b7c66b33f769fd"
server_key_version: "encryptionkey@0"
I20260927 15:14:22.343120 20029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 15:14:22.346661 20043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:22.348028 20029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:14:22.348191 20029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "0ab78f4caf814f26a025d6ef15d43d32"
format_stamp: "Formatted at 2026-09-27 15:14:22 on dist-test-slave-f5h6"
server_key: "a63b2956bcab4ef2bba0cb449b175b47"
server_key_iv: "a986bd9517cd807d14b7c66b33f769fd"
server_key_version: "encryptionkey@0"
I20260927 15:14:22.348392 20029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:14:22.371917 20029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:14:22.372786 20029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:14:22.373076 20029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:14:22.374125 20029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:14:22.375640 20029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:14:22.375735 20029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:22.375854 20029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:14:22.375921 20029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:22.406167 20029 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:32769
I20260927 15:14:22.406235 20155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:32769 every 8 connection(s)
I20260927 15:14:22.407610 20029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:14:22.410612 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 20029
I20260927 15:14:22.410794 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:14:22.411263 10373 external_mini_cluster.cc:1428] Setting key 8c11037c968164d8918ae16eb13d716d
I20260927 15:14:22.420383 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:22.410502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:48133 (local address 127.10.33.126:42211)
0927 15:14:22.410696 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:22.410702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:22.412215 (+ 1513us) server_negotiation.cc:407] Connection header received
0927 15:14:22.413993 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:22.413998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:22.414080 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:22.414166 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:22.415211 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:22.415923 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:22.417483 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:22.417964 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:22.418065 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:22.419550 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:22.419555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:22.419559 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:14:22.419584 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:22.419615 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:14:22.419628 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:22.419707 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:22.419944 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 15:14:22.420045 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 15:14:22.421501 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:22.411028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:42211 (local address 127.10.33.67:48133)
0927 15:14:22.412058 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:22.412103 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:22.413720 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:22.414403 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:22.414411 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:22.414506 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:22.415057 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:22.415073 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:22.416073 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:22.416077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:22.417352 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:22.417364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:22.418189 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:22.418197 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:22.418417 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:22.419087 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:14:22.419090 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:22.419146 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:14:22.419271 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:14:22.419719 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:14:22.419732 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:14:22.419741 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 15:14:22.419923 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 15:14:22.420360 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":310,"threads_started":1}
I20260927 15:14:22.422932 20156 heartbeater.cc:347] Connected to a master server at 127.10.33.126:42211
I20260927 15:14:22.423223 20156 heartbeater.cc:464] Registering TS with master...
I20260927 15:14:22.423975 20156 heartbeater.cc:511] Master 127.10.33.126:42211 requested a full tablet report, sending...
I20260927 15:14:22.425451 19709 ts_manager.cc:194] Registered new tserver with Master: 0ab78f4caf814f26a025d6ef15d43d32 (127.10.33.67:32769)
I20260927 15:14:22.426455 19709 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.33.67:48133
I20260927 15:14:22.426514 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 15:14:22.432688 19897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:22.431047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43392 (local address 127.10.33.126:42211)
0927 15:14:22.431302 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:22.431309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:22.431709 (+ 400us) server_negotiation.cc:407] Connection header received
0927 15:14:22.431896 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:22.431902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:22.431969 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:22.432051 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:22.432453 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43392: BlockingRecv error: recv got EOF from 127.0.0.1:43392 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
I20260927 15:14:22.433497 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 19768
I20260927 15:14:22.441335 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:22.434404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40256 (local address 127.10.33.65:34315)
0927 15:14:22.434914 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:22.434920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:22.434946 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:14:22.435006 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:22.435014 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:22.435224 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:22.435359 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:22.435675 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:22.436760 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:22.438062 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:22.438637 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:22.438733 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:22.439854 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:22.439864 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:22.439885 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:14:22.440475 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:22.440522 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:14:22.440543 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:22.440637 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:22.440832 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 15:14:22.440991 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":260,"threads_started":1}
I20260927 15:14:22.442633 19890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:14:22.524924 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 19768
I20260927 15:14:22.538033 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 19899
I20260927 15:14:22.545676 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:22.539000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50326 (local address 127.10.33.66:43763)
0927 15:14:22.539518 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:22.539525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:22.539551 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:14:22.539618 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:22.539624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:22.539836 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:22.539972 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:22.540342 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:22.541451 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:22.542749 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:22.543289 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:22.543386 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:22.544311 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:22.544319 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:22.544334 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:14:22.544918 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:22.544966 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:14:22.544989 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:22.545075 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:22.545198 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 15:14:22.545324 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":279,"threads_started":1}
I20260927 15:14:22.546984 20021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:14:22.628733 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 19899
I20260927 15:14:22.642444 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 20029
I20260927 15:14:22.651564 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:22.643527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39208 (local address 127.10.33.67:32769)
0927 15:14:22.644299 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:22.644315 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:22.644358 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:14:22.644425 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:22.644431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:22.644661 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:22.644791 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:22.645319 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:22.646523 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:22.647917 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:22.648513 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:22.648627 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:22.649976 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:22.649986 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:22.650002 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:14:22.650621 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:22.650672 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:14:22.650694 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:22.650791 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:22.651003 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 15:14:22.651125 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":394,"threads_started":1}
I20260927 15:14:22.652988 20151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:14:22.736377 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 20029
I20260927 15:14:22.749054 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 19678
I20260927 15:14:22.750340 19739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:14:22.849517 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 19678
2026-09-27T15:14:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2543 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T15:14:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:14:22Z Disabled control of system clock
I20260927 15:14:22.915871 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:45553
--webserver_interface=127.10.33.126
--webserver_port=0
--builtin_ntp_servers=127.10.33.84:36127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:45553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 15:14:23.249475 20175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:14:23.249800 20175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:14:23.249934 20175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:14:23.257350 20175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:14:23.257472 20175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:14:23.257508 20175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:14:23.257555 20175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:14:23.257598 20175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:14:23.267584 20175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:36127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:45553
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:45553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:14:23.268818 20175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:14:23.270612 20175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:14:23.282348 20180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:23.282616 20181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:23.282635 20183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:14:23.283475 20175 server_base.cc:1034] running on GCE node
I20260927 15:14:23.284314 20175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:14:23.285836 20175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:14:23.287144 20175 hybrid_clock.cc:648] HybridClock initialized: now 1790522063287088 us; error 44 us; skew 500 ppm
I20260927 15:14:23.287585 20175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:14:23.290117 20175 webserver.cc:460] Webserver started at http://127.10.33.126:37575/ using document root <none> and password file <none>
I20260927 15:14:23.291034 20175 fs_manager.cc:359] Metadata directory not provided
I20260927 15:14:23.291152 20175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:14:23.291577 20175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:14:23.296316 20175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "0d0bb42832d04b7e8c77abae15ab6944"
format_stamp: "Formatted at 2026-09-27 15:14:23 on dist-test-slave-f5h6"
server_key: "55cd4ca057a9defc3a9b64468bec7634"
server_key_iv: "43a3561dc554ee5266a5c8a2e6b512e2"
server_key_version: "encryptionkey@0"
I20260927 15:14:23.297169 20175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "0d0bb42832d04b7e8c77abae15ab6944"
format_stamp: "Formatted at 2026-09-27 15:14:23 on dist-test-slave-f5h6"
server_key: "55cd4ca057a9defc3a9b64468bec7634"
server_key_iv: "43a3561dc554ee5266a5c8a2e6b512e2"
server_key_version: "encryptionkey@0"
I20260927 15:14:23.304541 20175 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.005s
I20260927 15:14:23.308624 20189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:23.310148 20175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 15:14:23.310315 20175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "0d0bb42832d04b7e8c77abae15ab6944"
format_stamp: "Formatted at 2026-09-27 15:14:23 on dist-test-slave-f5h6"
server_key: "55cd4ca057a9defc3a9b64468bec7634"
server_key_iv: "43a3561dc554ee5266a5c8a2e6b512e2"
server_key_version: "encryptionkey@0"
I20260927 15:14:23.310536 20175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:14:23.316972 20175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:14:23.317822 20175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:14:23.318131 20175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:14:23.335558 20175 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:45553
I20260927 15:14:23.335590 20240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:45553 every 8 connection(s)
I20260927 15:14:23.336855 20175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:14:23.338784 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 20175
I20260927 15:14:23.338986 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:14:23.339758 10373 external_mini_cluster.cc:1428] Setting key 7fe7668a7d83f4d610b14e6ca1c65c1e
I20260927 15:14:23.341488 20241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:14:23.348852 20244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:23.341963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34950 (local address 127.10.33.126:45553)
0927 15:14:23.343053 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:23.343067 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:23.343117 (+ 50us) server_negotiation.cc:407] Connection header received
0927 15:14:23.344819 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:23.344841 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:23.345297 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:23.345666 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:23.346929 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:23.346937 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:23.346952 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:14:23.346969 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:23.347063 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:14:23.347081 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:23.347276 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:23.347520 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 15:14:23.347848 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":374,"threads_started":1}
I20260927 15:14:23.354738 20241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944: Bootstrap starting.
I20260927 15:14:23.358006 20241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944: Neither blocks nor log segments found. Creating new log.
I20260927 15:14:23.359035 20241 log.cc:824] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944: Log is configured to *not* fsync() on all Append() calls
I20260927 15:14:23.362291 20241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944: No bootstrap required, opened a new log
I20260927 15:14:23.369544 20241 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0bb42832d04b7e8c77abae15ab6944" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45553 } }
I20260927 15:14:23.369999 20241 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:14:23.370106 20241 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d0bb42832d04b7e8c77abae15ab6944, State: Initialized, Role: FOLLOWER
I20260927 15:14:23.370745 20241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0bb42832d04b7e8c77abae15ab6944" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45553 } }
I20260927 15:14:23.370981 20241 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:14:23.371104 20241 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:14:23.371259 20241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:14:23.373387 20241 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0bb42832d04b7e8c77abae15ab6944" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45553 } }
I20260927 15:14:23.373965 20241 leader_election.cc:302] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d0bb42832d04b7e8c77abae15ab6944; no voters:
I20260927 15:14:23.374517 20241 leader_election.cc:288] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:14:23.374657 20246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:14:23.375315 20246 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944 [term 1 LEADER]: Becoming Leader. State: Replica: 0d0bb42832d04b7e8c77abae15ab6944, State: Running, Role: LEADER
I20260927 15:14:23.375877 20246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0bb42832d04b7e8c77abae15ab6944" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45553 } }
I20260927 15:14:23.376665 20241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:14:23.378804 20248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d0bb42832d04b7e8c77abae15ab6944. Latest consensus state: current_term: 1 leader_uuid: "0d0bb42832d04b7e8c77abae15ab6944" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0bb42832d04b7e8c77abae15ab6944" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45553 } } }
I20260927 15:14:23.379211 20248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944 [sys.catalog]: This master's current role is: LEADER
I20260927 15:14:23.379026 20247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d0bb42832d04b7e8c77abae15ab6944" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0bb42832d04b7e8c77abae15ab6944" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45553 } } }
I20260927 15:14:23.379331 20247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944 [sys.catalog]: This master's current role is: LEADER
I20260927 15:14:23.380745 20255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:14:23.386053 20255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:14:23.395510 20255 catalog_manager.cc:1347] Generated new cluster ID: 542a1778b20f4e10951824500152ee31
I20260927 15:14:23.395633 20255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:14:23.449373 20255 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:14:23.449832 20255 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:14:23.460307 20255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d0bb42832d04b7e8c77abae15ab6944: Generated new TSK 0
I20260927 15:14:23.461149 20255 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:14:23.478034 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:45553
--builtin_ntp_servers=127.10.33.84:36127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 15:14:23.823573 20265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:14:23.823864 20265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:14:23.823961 20265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:14:23.832075 20265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:14:23.832207 20265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:14:23.832319 20265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:14:23.841945 20265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:36127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:45553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:14:23.843168 20265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:14:23.844878 20265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:14:23.858310 20273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:23.858237 20271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:23.858381 20270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:14:23.859277 20265 server_base.cc:1034] running on GCE node
I20260927 15:14:23.859905 20265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:14:23.860918 20265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:14:23.862131 20265 hybrid_clock.cc:648] HybridClock initialized: now 1790522063862080 us; error 37 us; skew 500 ppm
I20260927 15:14:23.862461 20265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:14:23.864786 20265 webserver.cc:460] Webserver started at http://127.10.33.65:46721/ using document root <none> and password file <none>
I20260927 15:14:23.865492 20265 fs_manager.cc:359] Metadata directory not provided
I20260927 15:14:23.865593 20265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:14:23.865924 20265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:14:23.868706 20265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "b733bafc63e447f7ba66851771387592"
format_stamp: "Formatted at 2026-09-27 15:14:23 on dist-test-slave-f5h6"
server_key: "43ea0116ce3e9720c076d601e34b9b55"
server_key_iv: "2231ff64c74ff79992aff7b32dcbedb1"
server_key_version: "encryptionkey@0"
I20260927 15:14:23.869481 20265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "b733bafc63e447f7ba66851771387592"
format_stamp: "Formatted at 2026-09-27 15:14:23 on dist-test-slave-f5h6"
server_key: "43ea0116ce3e9720c076d601e34b9b55"
server_key_iv: "2231ff64c74ff79992aff7b32dcbedb1"
server_key_version: "encryptionkey@0"
I20260927 15:14:23.874658 20265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 15:14:23.878541 20279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:23.879984 20265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:14:23.880162 20265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "b733bafc63e447f7ba66851771387592"
format_stamp: "Formatted at 2026-09-27 15:14:23 on dist-test-slave-f5h6"
server_key: "43ea0116ce3e9720c076d601e34b9b55"
server_key_iv: "2231ff64c74ff79992aff7b32dcbedb1"
server_key_version: "encryptionkey@0"
I20260927 15:14:23.880409 20265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:14:23.886466 20265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:14:23.887355 20265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:14:23.887609 20265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:14:23.888559 20265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:14:23.890182 20265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:14:23.890278 20265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:23.890372 20265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:14:23.890430 20265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:23.922608 20265 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:34387
I20260927 15:14:23.922667 20391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:34387 every 8 connection(s)
I20260927 15:14:23.923940 20265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:14:23.924749 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 20265
I20260927 15:14:23.924932 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:14:23.925698 10373 external_mini_cluster.cc:1428] Setting key 69c02b3ce414bd0aea5cfc2bc961b17f
I20260927 15:14:23.928455 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:45553
--builtin_ntp_servers=127.10.33.84:36127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 15:14:23.932780 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:23.926680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:51887 (local address 127.10.33.126:45553)
0927 15:14:23.927107 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:23.927111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:23.928587 (+ 1476us) server_negotiation.cc:407] Connection header received
0927 15:14:23.930437 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:23.930443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:23.930507 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:23.930595 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:23.932051 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:23.932056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:23.932059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:14:23.932077 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:23.932118 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:14:23.932130 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:23.932282 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:23.932424 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 15:14:23.932494 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":232,"threads_started":1}
I20260927 15:14:23.933799 20395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:23.927336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:45553 (local address 127.10.33.65:51887)
0927 15:14:23.928396 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:23.928439 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:23.930164 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:23.930851 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:23.930861 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:23.930959 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:23.931654 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:14:23.931657 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:23.931696 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:14:23.931871 (+ 175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:14:23.932217 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:14:23.932228 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:14:23.932235 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:14:23.932501 (+ 266us) client_negotiation.cc:240] Negotiation successful
0927 15:14:23.932796 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":311,"threads_started":1}
I20260927 15:14:23.935467 20392 heartbeater.cc:347] Connected to a master server at 127.10.33.126:45553
I20260927 15:14:23.935806 20392 heartbeater.cc:464] Registering TS with master...
I20260927 15:14:23.936488 20392 heartbeater.cc:511] Master 127.10.33.126:45553 requested a full tablet report, sending...
I20260927 15:14:23.938881 20206 ts_manager.cc:194] Registered new tserver with Master: b733bafc63e447f7ba66851771387592 (127.10.33.65:34387)
W20260927 15:14:24.268040 20396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:14:24.268332 20396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:14:24.268422 20396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:14:24.275800 20396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:14:24.275934 20396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:14:24.276046 20396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:14:24.285729 20396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:36127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:45553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:14:24.287182 20396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:14:24.289069 20396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:14:24.302496 20401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:24.302469 20404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:24.302691 20402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:14:24.303345 20396 server_base.cc:1034] running on GCE node
I20260927 15:14:24.304032 20396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:14:24.305152 20396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:14:24.306402 20396 hybrid_clock.cc:648] HybridClock initialized: now 1790522064306347 us; error 39 us; skew 500 ppm
I20260927 15:14:24.306769 20396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:14:24.309337 20396 webserver.cc:460] Webserver started at http://127.10.33.66:34599/ using document root <none> and password file <none>
I20260927 15:14:24.310133 20396 fs_manager.cc:359] Metadata directory not provided
I20260927 15:14:24.310266 20396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:14:24.310603 20396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:14:24.313795 20396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "8787d214589346079b1f42cd17431c13"
format_stamp: "Formatted at 2026-09-27 15:14:24 on dist-test-slave-f5h6"
server_key: "ccd531d36a1b782bfb72003daa578fd0"
server_key_iv: "f49127539915ebb5678cf60d7e35e72c"
server_key_version: "encryptionkey@0"
I20260927 15:14:24.314839 20396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "8787d214589346079b1f42cd17431c13"
format_stamp: "Formatted at 2026-09-27 15:14:24 on dist-test-slave-f5h6"
server_key: "ccd531d36a1b782bfb72003daa578fd0"
server_key_iv: "f49127539915ebb5678cf60d7e35e72c"
server_key_version: "encryptionkey@0"
I20260927 15:14:24.320823 20396 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 15:14:24.324430 20410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:24.326009 20396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:14:24.326174 20396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "8787d214589346079b1f42cd17431c13"
format_stamp: "Formatted at 2026-09-27 15:14:24 on dist-test-slave-f5h6"
server_key: "ccd531d36a1b782bfb72003daa578fd0"
server_key_iv: "f49127539915ebb5678cf60d7e35e72c"
server_key_version: "encryptionkey@0"
I20260927 15:14:24.326390 20396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:14:24.331991 20396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:14:24.332790 20396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:14:24.333024 20396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:14:24.333971 20396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:14:24.335464 20396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:14:24.335561 20396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:24.335662 20396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:14:24.335727 20396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:24.368914 20396 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:35251
I20260927 15:14:24.368958 20522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:35251 every 8 connection(s)
I20260927 15:14:24.370225 20396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:14:24.373265 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 20396
I20260927 15:14:24.373442 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:14:24.374081 10373 external_mini_cluster.cc:1428] Setting key e6ff1bf940315201d1582a17807da5fa
I20260927 15:14:24.376811 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:45553
--builtin_ntp_servers=127.10.33.84:36127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 15:14:24.378880 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:24.372730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:49955 (local address 127.10.33.126:45553)
0927 15:14:24.372920 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:24.372925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:24.374606 (+ 1681us) server_negotiation.cc:407] Connection header received
0927 15:14:24.376354 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:24.376359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:24.376446 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:24.376558 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:24.378202 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:24.378208 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:24.378213 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:14:24.378234 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:24.378273 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:14:24.378287 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:24.378416 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:24.378559 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 15:14:24.378623 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 15:14:24.379988 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:24.373327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:45553 (local address 127.10.33.66:49955)
0927 15:14:24.374439 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:24.374481 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:24.376045 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:24.376843 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:24.376853 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:24.377072 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:24.377814 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:14:24.377817 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:24.377896 (+ 79us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:14:24.378020 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:14:24.378379 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:14:24.378385 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:14:24.378390 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:14:24.378619 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 15:14:24.378866 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":339,"threads_started":1}
I20260927 15:14:24.381306 20523 heartbeater.cc:347] Connected to a master server at 127.10.33.126:45553
I20260927 15:14:24.381610 20523 heartbeater.cc:464] Registering TS with master...
I20260927 15:14:24.382272 20523 heartbeater.cc:511] Master 127.10.33.126:45553 requested a full tablet report, sending...
I20260927 15:14:24.383742 20206 ts_manager.cc:194] Registered new tserver with Master: 8787d214589346079b1f42cd17431c13 (127.10.33.66:35251)
W20260927 15:14:24.706300 20526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:14:24.706614 20526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:14:24.706732 20526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:14:24.714874 20526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:14:24.714998 20526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:14:24.715142 20526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:14:24.724445 20526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:36127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:45553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:14:24.725795 20526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:14:24.727618 20526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:14:24.741029 20531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:24.741007 20534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:24.740988 20532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:14:24.741938 20526 server_base.cc:1034] running on GCE node
I20260927 15:14:24.742547 20526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:14:24.743494 20526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:14:24.744714 20526 hybrid_clock.cc:648] HybridClock initialized: now 1790522064744656 us; error 51 us; skew 500 ppm
I20260927 15:14:24.745013 20526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:14:24.753131 20526 webserver.cc:460] Webserver started at http://127.10.33.67:37353/ using document root <none> and password file <none>
I20260927 15:14:24.753839 20526 fs_manager.cc:359] Metadata directory not provided
I20260927 15:14:24.754091 20526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:14:24.754401 20526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:14:24.757273 20526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "512cfe6e7c784539a6abc0c0e47ad9ed"
format_stamp: "Formatted at 2026-09-27 15:14:24 on dist-test-slave-f5h6"
server_key: "ac6a7a9c716857e750377fb99fa15132"
server_key_iv: "12a489f9fa7805097d86f8aa512e161b"
server_key_version: "encryptionkey@0"
I20260927 15:14:24.758038 20526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "512cfe6e7c784539a6abc0c0e47ad9ed"
format_stamp: "Formatted at 2026-09-27 15:14:24 on dist-test-slave-f5h6"
server_key: "ac6a7a9c716857e750377fb99fa15132"
server_key_iv: "12a489f9fa7805097d86f8aa512e161b"
server_key_version: "encryptionkey@0"
I20260927 15:14:24.763245 20526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260927 15:14:24.767038 20540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:24.768781 20526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:14:24.769001 20526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "512cfe6e7c784539a6abc0c0e47ad9ed"
format_stamp: "Formatted at 2026-09-27 15:14:24 on dist-test-slave-f5h6"
server_key: "ac6a7a9c716857e750377fb99fa15132"
server_key_iv: "12a489f9fa7805097d86f8aa512e161b"
server_key_version: "encryptionkey@0"
I20260927 15:14:24.769296 20526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:14:24.775290 20526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:14:24.776197 20526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:14:24.776474 20526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:14:24.777503 20526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:14:24.779027 20526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:14:24.779141 20526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:24.779263 20526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:14:24.779337 20526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:24.812525 20526 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:41517
I20260927 15:14:24.812600 20652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:41517 every 8 connection(s)
I20260927 15:14:24.813972 20526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:14:24.820308 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 20526
I20260927 15:14:24.820458 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:14:24.820984 10373 external_mini_cluster.cc:1428] Setting key 864050b65b427dcd7a1d5593b58b7b18
I20260927 15:14:24.822167 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:24.816604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:53081 (local address 127.10.33.126:45553)
0927 15:14:24.816793 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:24.816800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:24.818295 (+ 1495us) server_negotiation.cc:407] Connection header received
0927 15:14:24.819963 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:24.819969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:24.820047 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:24.820165 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:24.821489 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:24.821494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:24.821497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:14:24.821513 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:24.821545 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:14:24.821556 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:24.821655 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:24.821840 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 15:14:24.821920 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:14:24.823447 20655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:24.817132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:45553 (local address 127.10.33.67:53081)
0927 15:14:24.818150 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:24.818187 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:24.819663 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:24.820347 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:24.820357 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:24.820457 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:24.821180 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:14:24.821182 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:24.821220 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:14:24.821307 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:14:24.821649 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:14:24.821660 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:14:24.821668 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 15:14:24.821945 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 15:14:24.822224 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":291,"threads_started":1}
I20260927 15:14:24.824784 20653 heartbeater.cc:347] Connected to a master server at 127.10.33.126:45553
I20260927 15:14:24.825137 20653 heartbeater.cc:464] Registering TS with master...
I20260927 15:14:24.825937 20653 heartbeater.cc:511] Master 127.10.33.126:45553 requested a full tablet report, sending...
I20260927 15:14:24.827456 20206 ts_manager.cc:194] Registered new tserver with Master: 512cfe6e7c784539a6abc0c0e47ad9ed (127.10.33.67:41517)
I20260927 15:14:24.837088 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 15:14:24.843024 20394 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:14:24.841485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34956 (local address 127.10.33.126:45553)
0927 15:14:24.841632 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:24.841636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:24.842186 (+ 550us) server_negotiation.cc:407] Connection header received
0927 15:14:24.842323 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:24.842327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:24.842402 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:24.842489 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:24.842841 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34956: BlockingRecv error: recv got EOF from 127.0.0.1:34956 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:14:24.843724 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 20265
I20260927 15:14:24.846935 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:24.844571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54654 (local address 127.10.33.65:34387)
0927 15:14:24.845046 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:24.845053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:24.845078 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:14:24.845139 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:24.845145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:24.845351 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:24.845493 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:24.846392 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:24.846398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:24.846408 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:14:24.846431 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:24.846467 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:14:24.846481 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:24.846578 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:24.846644 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 15:14:24.846716 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":237,"threads_started":1}
I20260927 15:14:24.848083 20387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:14:24.929495 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 20265
I20260927 15:14:24.943859 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 20396
I20260927 15:14:24.947569 20664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:24.944829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49430 (local address 127.10.33.66:35251)
0927 15:14:24.945391 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:24.945398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:24.945426 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:14:24.945501 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:24.945508 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:24.945729 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:24.945906 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:24.946860 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:24.946867 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:24.946878 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:14:24.946896 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:24.946935 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:14:24.946948 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:24.947070 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:24.947204 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 15:14:24.947310 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":284,"threads_started":1}
I20260927 15:14:24.949050 20518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:14:25.029439 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 20396
I20260927 15:14:25.042649 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 20526
I20260927 15:14:25.047032 20666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:25.043727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39968 (local address 127.10.33.67:41517)
0927 15:14:25.044432 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:25.044439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:25.044477 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:14:25.044537 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:25.044544 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:25.044756 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:25.044981 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:25.046274 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:25.046281 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:25.046293 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:14:25.046310 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:25.046350 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:14:25.046364 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:25.046525 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:25.046626 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 15:14:25.046736 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":390,"threads_started":1}
I20260927 15:14:25.048233 20648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:14:25.130896 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 20526
I20260927 15:14:25.143265 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 20175
I20260927 15:14:25.144465 20236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:14:25.243896 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 20175
2026-09-27T15:14:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2395 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:14:25 dist-test-slave-f5h6 krb5kdc[20671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:14:25 dist-test-slave-f5h6 krb5kdc[20671](info): set up 2 sockets
Sep 27 15:14:25 dist-test-slave-f5h6 krb5kdc[20671](info): commencing operation
krb5kdc: starting...
W20260927 15:14:26.294665 10373 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:14:26 dist-test-slave-f5h6 krb5kdc[20671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:14:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:14:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:14:26.527134 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:45061
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:33847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:45061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:14:26.864708 20684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:14:26.865091 20684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:14:26.865186 20684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:14:26.872609 20684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:14:26.872730 20684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:14:26.872777 20684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:14:26.872833 20684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:14:26.872875 20684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:14:26.882879 20684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:33847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:45061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:45061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:14:26.884160 20684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:14:26.885433 20684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:14:26.898257 20690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:26.898327 20689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:26.898325 20692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:14:26.899140 20684 server_base.cc:1034] running on GCE node
I20260927 15:14:26.899919 20684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:14:26.901425 20684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:14:26.902657 20684 hybrid_clock.cc:648] HybridClock initialized: now 1790522066902603 us; error 38 us; skew 500 ppm
I20260927 15:14:26.903028 20684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:14:26 dist-test-slave-f5h6 krb5kdc[20671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522066, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:14:26.906191 20684 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:14:26.907702 20684 webserver.cc:460] Webserver started at http://127.10.33.126:42347/ using document root <none> and password file <none>
I20260927 15:14:26.908358 20684 fs_manager.cc:359] Metadata directory not provided
I20260927 15:14:26.908483 20684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:14:26.908764 20684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:14:26.911770 20684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "38ed0679a3484d48afeb662cb2e588bd"
format_stamp: "Formatted at 2026-09-27 15:14:26 on dist-test-slave-f5h6"
server_key: "222ab2f9b51413ac41a5e07eeac3d53b"
server_key_iv: "dba66791d2a7d45a03cb1e8740ec904f"
server_key_version: "encryptionkey@0"
I20260927 15:14:26.912667 20684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "38ed0679a3484d48afeb662cb2e588bd"
format_stamp: "Formatted at 2026-09-27 15:14:26 on dist-test-slave-f5h6"
server_key: "222ab2f9b51413ac41a5e07eeac3d53b"
server_key_iv: "dba66791d2a7d45a03cb1e8740ec904f"
server_key_version: "encryptionkey@0"
I20260927 15:14:26.918104 20684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 15:14:26.922057 20699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:26.923578 20684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:14:26.923781 20684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "38ed0679a3484d48afeb662cb2e588bd"
format_stamp: "Formatted at 2026-09-27 15:14:26 on dist-test-slave-f5h6"
server_key: "222ab2f9b51413ac41a5e07eeac3d53b"
server_key_iv: "dba66791d2a7d45a03cb1e8740ec904f"
server_key_version: "encryptionkey@0"
I20260927 15:14:26.924089 20684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:14:26.945065 20684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:14:26.950002 20684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:14:26.950284 20684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:14:26.968751 20684 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:45061
I20260927 15:14:26.968780 20750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:45061 every 8 connection(s)
I20260927 15:14:26.970135 20684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:14:26.973822 20751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:14:26.980378 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 20684
I20260927 15:14:26.980613 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:14:26.981397 10373 external_mini_cluster.cc:1428] Setting key 080098d39f3e39866b8fca54c0e9ff11
I20260927 15:14:26.986312 20751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd: Bootstrap starting.
I20260927 15:14:26.990247 20751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd: Neither blocks nor log segments found. Creating new log.
I20260927 15:14:26.991639 20751 log.cc:824] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd: Log is configured to *not* fsync() on all Append() calls
I20260927 15:14:26.995173 20751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd: No bootstrap required, opened a new log
Sep 27 15:14:26 dist-test-slave-f5h6 krb5kdc[20671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:14:27.004257 20751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ed0679a3484d48afeb662cb2e588bd" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45061 } }
I20260927 15:14:27.004771 20751 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:14:27.004938 20751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ed0679a3484d48afeb662cb2e588bd, State: Initialized, Role: FOLLOWER
I20260927 15:14:27.005658 20751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ed0679a3484d48afeb662cb2e588bd" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45061 } }
I20260927 15:14:27.005996 20751 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:14:27.006136 20751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:14:27.006304 20751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:14:27.008709 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:26.983687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40362 (local address 127.10.33.126:45061)
0927 15:14:26.984524 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:26.984539 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:26.984592 (+ 53us) server_negotiation.cc:407] Connection header received
0927 15:14:26.986954 (+ 2362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:26.986984 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:26.987554 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:26.988043 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:26.989697 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:26.991825 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:26.993367 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:26.994415 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:26.994614 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:26.999196 (+ 4582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:26.999205 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:26.999224 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:14:26.999270 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:27.002951 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:27.004013 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:27.004021 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:27.004028 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:27.004230 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:27.004713 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:27.004719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:27.004722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:27.005431 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:27.005721 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:27.006733 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0927 15:14:27.007197 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":341,"threads_started":1}
I20260927 15:14:27.009073 20751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ed0679a3484d48afeb662cb2e588bd" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45061 } }
I20260927 15:14:27.009639 20751 leader_election.cc:302] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38ed0679a3484d48afeb662cb2e588bd; no voters:
I20260927 15:14:27.010290 20751 leader_election.cc:288] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:14:27.010578 20756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:14:27.011919 20756 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd [term 1 LEADER]: Becoming Leader. State: Replica: 38ed0679a3484d48afeb662cb2e588bd, State: Running, Role: LEADER
I20260927 15:14:27.012496 20756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ed0679a3484d48afeb662cb2e588bd" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45061 } }
I20260927 15:14:27.013332 20751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:14:27.015523 20757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38ed0679a3484d48afeb662cb2e588bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ed0679a3484d48afeb662cb2e588bd" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45061 } } }
I20260927 15:14:27.016019 20757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd [sys.catalog]: This master's current role is: LEADER
I20260927 15:14:27.016507 20758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38ed0679a3484d48afeb662cb2e588bd. Latest consensus state: current_term: 1 leader_uuid: "38ed0679a3484d48afeb662cb2e588bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ed0679a3484d48afeb662cb2e588bd" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45061 } } }
I20260927 15:14:27.016956 20758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd [sys.catalog]: This master's current role is: LEADER
I20260927 15:14:27.022861 20765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:14:27.027944 20765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:14:27.037398 20765 catalog_manager.cc:1347] Generated new cluster ID: 061afe05bce44eab886d7bc10a11edaa
I20260927 15:14:27.037546 20765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:14:27.052037 20765 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:14:27.053634 20765 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:14:27.066570 20765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd: Generated new TSK 0
I20260927 15:14:27.067507 20765 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.33.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
I20260927 15:14:27.135349 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:45061
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:33847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:14:27.502431 20777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:14:27.502743 20777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:14:27.502837 20777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:14:27.512202 20777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:14:27.512347 20777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:14:27.512471 20777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:14:27.524096 20777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:33847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:45061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:14:27.525499 20777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:14:27.526917 20777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:14:27.541162 20782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:27.541054 20783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:27.541105 20785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:14:27.542212 20777 server_base.cc:1034] running on GCE node
I20260927 15:14:27.542999 20777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:14:27.544031 20777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:14:27.545295 20777 hybrid_clock.cc:648] HybridClock initialized: now 1790522067545201 us; error 59 us; skew 500 ppm
I20260927 15:14:27.545644 20777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:14:27 dist-test-slave-f5h6 krb5kdc[20671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522067, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:14:27.549207 20777 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:14:27.550809 20777 webserver.cc:460] Webserver started at http://127.10.33.65:36717/ using document root <none> and password file <none>
I20260927 15:14:27.551539 20777 fs_manager.cc:359] Metadata directory not provided
I20260927 15:14:27.551662 20777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:14:27.551967 20777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:14:27.555743 20777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "6840f5ea9ca2448b989c3d0a24767706"
format_stamp: "Formatted at 2026-09-27 15:14:27 on dist-test-slave-f5h6"
server_key: "acb02af942f6b0a3dc67bda0509543ec"
server_key_iv: "5f5a13eeeb8c699e689b203427f6179e"
server_key_version: "encryptionkey@0"
I20260927 15:14:27.556579 20777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "6840f5ea9ca2448b989c3d0a24767706"
format_stamp: "Formatted at 2026-09-27 15:14:27 on dist-test-slave-f5h6"
server_key: "acb02af942f6b0a3dc67bda0509543ec"
server_key_iv: "5f5a13eeeb8c699e689b203427f6179e"
server_key_version: "encryptionkey@0"
I20260927 15:14:27.562489 20777 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260927 15:14:27.566361 20792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:27.568005 20777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:14:27.568193 20777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "6840f5ea9ca2448b989c3d0a24767706"
format_stamp: "Formatted at 2026-09-27 15:14:27 on dist-test-slave-f5h6"
server_key: "acb02af942f6b0a3dc67bda0509543ec"
server_key_iv: "5f5a13eeeb8c699e689b203427f6179e"
server_key_version: "encryptionkey@0"
I20260927 15:14:27.568390 20777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:14:27.581393 20777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:14:27.586477 20777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:14:27.586792 20777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:14:27.587833 20777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:14:27.589359 20777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:14:27.589448 20777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:27.589570 20777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:14:27.589620 20777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:27.624123 20777 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:33371
I20260927 15:14:27.624130 20904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:33371 every 8 connection(s)
I20260927 15:14:27.625504 20777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:14:27.630446 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 20777
I20260927 15:14:27.630659 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:14:27.631502 10373 external_mini_cluster.cc:1428] Setting key 869a00d368dc9a89f64d978a7abf69c6
Sep 27 15:14:27 dist-test-slave-f5h6 krb5kdc[20671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522067, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:14:27.649847 20907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:27.628524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:55221 (local address 127.10.33.126:45061)
0927 15:14:27.629080 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:27.629085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:27.630154 (+ 1069us) server_negotiation.cc:407] Connection header received
0927 15:14:27.632484 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:27.632489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:27.632575 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:27.632669 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:27.634558 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:27.635284 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:27.637370 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:27.637762 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:27.637909 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:27.642040 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:27.642048 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:27.642053 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:14:27.642098 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:27.645450 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:27.646782 (+ 1332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:27.646788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:27.646790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:27.646932 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:27.647517 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:27.647522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:27.647524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:27.647874 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:27.648002 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:27.649150 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0927 15:14:27.649379 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":241,"threads_started":1}
I20260927 15:14:27.651325 20908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:27.629180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:45061 (local address 127.10.33.65:55221)
0927 15:14:27.629980 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:27.630030 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:27.632212 (+ 2182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:27.632910 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:27.632919 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:27.633533 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:27.634361 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:27.634386 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:27.635455 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:27.635460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:27.637183 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:27.637197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:27.638016 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:27.638022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:27.638305 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:27.639072 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:27.639099 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:27.641722 (+ 2623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:14:27.645721 (+ 3999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:14:27.645737 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:14:27.645760 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:14:27.646567 (+ 807us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:14:27.647047 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:14:27.647052 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:14:27.647055 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:14:27.647343 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:14:27.648012 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:14:27.648020 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:14:27.648518 (+ 498us) client_negotiation.cc:769] Sending connection context
0927 15:14:27.648823 (+ 305us) client_negotiation.cc:240] Negotiation successful
0927 15:14:27.649535 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":295,"threads_started":1}
I20260927 15:14:27.653352 20905 heartbeater.cc:347] Connected to a master server at 127.10.33.126:45061
I20260927 15:14:27.653733 20905 heartbeater.cc:464] Registering TS with master...
I20260927 15:14:27.654771 20905 heartbeater.cc:511] Master 127.10.33.126:45061 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.33.66@KRBTEST.COM; defaulting to no policy
I20260927 15:14:27.657730 20716 ts_manager.cc:194] Registered new tserver with Master: 6840f5ea9ca2448b989c3d0a24767706 (127.10.33.65:33371)
I20260927 15:14:27.659351 20716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:55221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
I20260927 15:14:27.698273 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:45061
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:33847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:14:28.045514 20911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:14:28.045827 20911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:14:28.045972 20911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:14:28.053190 20911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:14:28.053331 20911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:14:28.053495 20911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:14:28.063786 20911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:33847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:45061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:14:28.065114 20911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:14:28.066633 20911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:14:28.079991 20916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:28.080014 20919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:28.080369 20917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:14:28.081360 20911 server_base.cc:1034] running on GCE node
I20260927 15:14:28.082144 20911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:14:28.083151 20911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:14:28.084375 20911 hybrid_clock.cc:648] HybridClock initialized: now 1790522068084320 us; error 52 us; skew 500 ppm
I20260927 15:14:28.084712 20911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:14:28 dist-test-slave-f5h6 krb5kdc[20671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522068, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:14:28.088096 20911 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:14:28.089589 20911 webserver.cc:460] Webserver started at http://127.10.33.66:32785/ using document root <none> and password file <none>
I20260927 15:14:28.090337 20911 fs_manager.cc:359] Metadata directory not provided
I20260927 15:14:28.090441 20911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:14:28.090719 20911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:14:28.093526 20911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "14573af1c7f049918d56ae41cdd32f08"
format_stamp: "Formatted at 2026-09-27 15:14:28 on dist-test-slave-f5h6"
server_key: "a12237796c2705ce136f76fe3f8de47f"
server_key_iv: "7a5fe665f4ed14c4a6ea7784a55f23cd"
server_key_version: "encryptionkey@0"
I20260927 15:14:28.094352 20911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "14573af1c7f049918d56ae41cdd32f08"
format_stamp: "Formatted at 2026-09-27 15:14:28 on dist-test-slave-f5h6"
server_key: "a12237796c2705ce136f76fe3f8de47f"
server_key_iv: "7a5fe665f4ed14c4a6ea7784a55f23cd"
server_key_version: "encryptionkey@0"
I20260927 15:14:28.099702 20911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:14:28.103587 20926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:28.105211 20911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:14:28.105397 20911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "14573af1c7f049918d56ae41cdd32f08"
format_stamp: "Formatted at 2026-09-27 15:14:28 on dist-test-slave-f5h6"
server_key: "a12237796c2705ce136f76fe3f8de47f"
server_key_iv: "7a5fe665f4ed14c4a6ea7784a55f23cd"
server_key_version: "encryptionkey@0"
I20260927 15:14:28.105626 20911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:14:28.128615 20911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:14:28.133478 20911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:14:28.133788 20911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:14:28.134955 20911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:14:28.136533 20911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:14:28.136616 20911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:28.136741 20911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:14:28.136792 20911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:28.172366 20911 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:36411
I20260927 15:14:28.172379 21038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:36411 every 8 connection(s)
I20260927 15:14:28.173776 20911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:14:28.182631 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 20911
I20260927 15:14:28.182886 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:14:28.183493 10373 external_mini_cluster.cc:1428] Setting key 8b081d53460d2fe439455cd415a7ce55
Sep 27 15:14:28 dist-test-slave-f5h6 krb5kdc[20671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522068, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:14:28.198166 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:28.176528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:59559 (local address 127.10.33.126:45061)
0927 15:14:28.177026 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:28.177032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:28.178040 (+ 1008us) server_negotiation.cc:407] Connection header received
0927 15:14:28.179833 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:28.179838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:28.179909 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:28.180045 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:28.181811 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:28.182705 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:28.184488 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:28.185120 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:28.185270 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:28.190160 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:28.190167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:28.190171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:14:28.190219 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:28.194017 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:28.195200 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:28.195205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:28.195208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:28.195363 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:28.195905 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:28.195910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:28.195914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:28.196309 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:28.196447 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:28.197586 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0927 15:14:28.197816 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":240,"threads_started":1}
I20260927 15:14:28.199313 21042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:28.177016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:45061 (local address 127.10.33.66:59559)
0927 15:14:28.177816 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:28.177906 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:28.179546 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:28.180275 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:28.180284 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:28.180986 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:28.181596 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:28.181616 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:28.182884 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:28.182888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:28.184279 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:28.184294 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:28.185893 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:28.185907 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:28.186163 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:28.186910 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:28.186942 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:28.189797 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:14:28.194244 (+ 4447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:14:28.194256 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:14:28.194278 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:14:28.194909 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:14:28.195492 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:14:28.195497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:14:28.195501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:14:28.195755 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:14:28.196443 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:14:28.196452 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:14:28.196960 (+ 508us) client_negotiation.cc:769] Sending connection context
0927 15:14:28.197268 (+ 308us) client_negotiation.cc:240] Negotiation successful
0927 15:14:28.197846 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":337,"threads_started":1}
I20260927 15:14:28.200939 21039 heartbeater.cc:347] Connected to a master server at 127.10.33.126:45061
I20260927 15:14:28.201270 21039 heartbeater.cc:464] Registering TS with master...
I20260927 15:14:28.202108 21039 heartbeater.cc:511] Master 127.10.33.126:45061 requested a full tablet report, sending...
I20260927 15:14:28.203729 20716 ts_manager.cc:194] Registered new tserver with Master: 14573af1c7f049918d56ae41cdd32f08 (127.10.33.66:36411)
I20260927 15:14:28.204798 20716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:59559
WARNING: no policy specified for kudu/127.10.33.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
I20260927 15:14:28.238812 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:45061
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:33847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:14:28.581255 21045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:14:28.581579 21045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:14:28.581676 21045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:14:28.589368 21045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:14:28.589509 21045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:14:28.589625 21045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:14:28.601697 21045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:33847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:45061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:14:28.603338 21045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:14:28.604704 21045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:14:28.620081 21050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:28.620816 21053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:14:28.620941 21051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:14:28.621212 21045 server_base.cc:1034] running on GCE node
I20260927 15:14:28.621991 21045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:14:28.623167 21045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:14:28.624454 21045 hybrid_clock.cc:648] HybridClock initialized: now 1790522068624323 us; error 117 us; skew 500 ppm
I20260927 15:14:28.624822 21045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:14:28 dist-test-slave-f5h6 krb5kdc[20671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522068, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:14:28.628237 21045 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:14:28.629920 21045 webserver.cc:460] Webserver started at http://127.10.33.67:35051/ using document root <none> and password file <none>
I20260927 15:14:28.630616 21045 fs_manager.cc:359] Metadata directory not provided
I20260927 15:14:28.630734 21045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:14:28.631026 21045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:14:28.633986 21045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "bbcead55eaa742d584f24d36afd99501"
format_stamp: "Formatted at 2026-09-27 15:14:28 on dist-test-slave-f5h6"
server_key: "0c924f30918b364eeafff76cb6266b5b"
server_key_iv: "5c755d3407895314a0e60976b362ba1a"
server_key_version: "encryptionkey@0"
I20260927 15:14:28.634706 21045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "bbcead55eaa742d584f24d36afd99501"
format_stamp: "Formatted at 2026-09-27 15:14:28 on dist-test-slave-f5h6"
server_key: "0c924f30918b364eeafff76cb6266b5b"
server_key_iv: "5c755d3407895314a0e60976b362ba1a"
server_key_version: "encryptionkey@0"
I20260927 15:14:28.639967 21045 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:14:28.643687 21060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:28.645346 21045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:14:28.645540 21045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "bbcead55eaa742d584f24d36afd99501"
format_stamp: "Formatted at 2026-09-27 15:14:28 on dist-test-slave-f5h6"
server_key: "0c924f30918b364eeafff76cb6266b5b"
server_key_iv: "5c755d3407895314a0e60976b362ba1a"
server_key_version: "encryptionkey@0"
I20260927 15:14:28.645737 21045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:14:28.663013 20905 heartbeater.cc:503] Master 127.10.33.126:45061 was elected leader, sending a full tablet report...
I20260927 15:14:28.672482 21045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:14:28.677687 21045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:14:28.678087 21045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:14:28.679354 21045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:14:28.681375 21045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:14:28.681624 21045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:14:28.681829 21045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:14:28.681910 21045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 15:14:28.714808 21045 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:41625
I20260927 15:14:28.714923 21172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:41625 every 8 connection(s)
I20260927 15:14:28.716131 21045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:14:28.723181 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 21045
I20260927 15:14:28.723551 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:14:28.724116 10373 external_mini_cluster.cc:1428] Setting key 26b8651abba11c64c0d5dd469c0c4171
Sep 27 15:14:28 dist-test-slave-f5h6 krb5kdc[20671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522068, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:14:28.740646 21175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:28.718805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:33619 (local address 127.10.33.126:45061)
0927 15:14:28.719477 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:28.719483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:28.720481 (+ 998us) server_negotiation.cc:407] Connection header received
0927 15:14:28.722733 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:28.722739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:28.722822 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:28.722933 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:28.725222 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:28.726218 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:28.728431 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:28.728911 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:28.729031 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:28.733501 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:28.733507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:28.733511 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:14:28.733554 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:28.736557 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:28.737569 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:28.737574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:28.737577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:28.737717 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:28.738400 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:28.738405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:28.738407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:28.738759 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:28.738908 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:28.740043 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0927 15:14:28.740257 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":332,"threads_started":1}
I20260927 15:14:28.741711 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:28.719413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:45061 (local address 127.10.33.67:33619)
0927 15:14:28.720263 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:28.720304 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:28.722116 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:28.723279 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:28.723294 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:28.724233 (+ 939us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:14:28.725041 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:28.725064 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:28.726400 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:28.726406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:28.728210 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:28.728224 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:28.729333 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:28.729342 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:28.729659 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:28.730615 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:14:28.730643 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:14:28.733173 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:14:28.736787 (+ 3614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:14:28.736795 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:14:28.736814 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:14:28.737401 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:14:28.737987 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:14:28.737991 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:14:28.737994 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:14:28.738206 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:14:28.738959 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:14:28.738969 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:14:28.739410 (+ 441us) client_negotiation.cc:769] Sending connection context
0927 15:14:28.739865 (+ 455us) client_negotiation.cc:240] Negotiation successful
0927 15:14:28.740452 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":294,"threads_started":1}
I20260927 15:14:28.743674 21173 heartbeater.cc:347] Connected to a master server at 127.10.33.126:45061
I20260927 15:14:28.744011 21173 heartbeater.cc:464] Registering TS with master...
I20260927 15:14:28.744735 21173 heartbeater.cc:511] Master 127.10.33.126:45061 requested a full tablet report, sending...
I20260927 15:14:28.746387 20716 ts_manager.cc:194] Registered new tserver with Master: bbcead55eaa742d584f24d36afd99501 (127.10.33.67:41625)
I20260927 15:14:28.747179 20716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:33619
I20260927 15:14:28.753284 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:14:28.772179 21175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:28.757974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40376 (local address 127.10.33.126:45061)
0927 15:14:28.758248 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:28.758255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:28.758697 (+ 442us) server_negotiation.cc:407] Connection header received
0927 15:14:28.758817 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:28.758823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:28.758943 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:28.759209 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:28.760819 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:28.761556 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:28.763306 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:28.763730 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:28.763872 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:28.766977 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:28.766982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:28.766986 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:14:28.767033 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:28.769601 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:28.770337 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:28.770342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:28.770344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:28.770472 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:28.770858 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:28.770861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:28.770863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:28.771213 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:28.771392 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:28.771718 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 15:14:28.771898 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:14:28.776324 20716 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:14:28.783313 20716 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:14:28.808936 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:28.796804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:41625 (local address 127.0.0.1:54812)
0927 15:14:28.798736 (+ 1932us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:28.798787 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:28.799007 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:28.800586 (+ 1579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:28.800594 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:28.800626 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:14:28.801090 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:28.801104 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:28.803353 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:28.803358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:28.805934 (+ 2576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:28.805958 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:28.807898 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:28.807905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:28.808109 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:28.808167 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 15:14:28.808287 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 15:14:28.808480 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1633,"thread_start_us":2867,"threads_started":1}
I20260927 15:14:28.810434 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:28.797043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54812 (local address 127.10.33.67:41625)
0927 15:14:28.799652 (+ 2609us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:28.799661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:28.799696 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:14:28.799795 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:28.799803 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:28.800334 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:28.801306 (+ 972us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:14:28.801599 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:28.803205 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:28.806157 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:28.807637 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:28.809492 (+ 1855us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:28.809755 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:28.809922 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 15:14:28.810109 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2449,"thread_start_us":2294,"threads_started":1}
I20260927 15:14:28.813071 21186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:28.797257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:36411 (local address 127.0.0.1:50422)
0927 15:14:28.799930 (+ 2673us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:28.799948 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:28.800057 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:28.802478 (+ 2421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:28.802484 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:28.802511 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:14:28.802911 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:28.802923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:28.806411 (+ 3488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:28.806417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:28.809768 (+ 3351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:28.809784 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:28.812240 (+ 2456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:28.812250 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:28.812459 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:28.812494 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 15:14:28.812592 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 15:14:28.812740 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2558,"thread_start_us":1012,"threads_started":1}
I20260927 15:14:28.813840 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:28.797781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50422 (local address 127.10.33.66:36411)
0927 15:14:28.801818 (+ 4037us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:28.801827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:28.801879 (+ 52us) server_negotiation.cc:407] Connection header received
0927 15:14:28.801982 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:28.801990 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:28.802252 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:28.803102 (+ 850us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:14:28.803312 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:28.806248 (+ 2936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:28.810004 (+ 3756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:28.811702 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:28.811848 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:28.812052 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:28.813328 (+ 1276us) server_negotiation.cc:299] Negotiation successful
0927 15:14:28.813528 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3871,"thread_start_us":3740,"threads_started":1}
I20260927 15:14:28.816674 21108 tablet_service.cc:1467] Processing CreateTablet for tablet 560011c1af4f4bdbb2da18b768863bfc (DEFAULT_TABLE table=test-table [id=160550ef7fe84ac3a42373173eb9c7ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:14:28.817402 20974 tablet_service.cc:1467] Processing CreateTablet for tablet 560011c1af4f4bdbb2da18b768863bfc (DEFAULT_TABLE table=test-table [id=160550ef7fe84ac3a42373173eb9c7ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:14:28.818284 21108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 560011c1af4f4bdbb2da18b768863bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:14:28.818435 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:28.800039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33371 (local address 127.0.0.1:35478)
0927 15:14:28.809202 (+ 9163us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:28.809223 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:28.809313 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:28.811886 (+ 2573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:28.811892 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:28.811930 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:14:28.812158 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:28.812172 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:28.814893 (+ 2721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:28.814899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:28.816135 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:28.816150 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:28.817620 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:28.817626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:28.817836 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:28.817902 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 15:14:28.817987 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 15:14:28.818160 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9068,"thread_start_us":12172,"threads_started":1}
I20260927 15:14:28.818503 21191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:28.800358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35478 (local address 127.10.33.65:33371)
0927 15:14:28.811188 (+ 10830us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:28.811197 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:28.811230 (+ 33us) server_negotiation.cc:407] Connection header received
0927 15:14:28.811333 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:28.811341 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:28.811626 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:28.811875 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:14:28.812376 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:28.814759 (+ 2383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:28.816305 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:28.817413 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:28.817588 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:28.817816 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:28.818051 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 15:14:28.818206 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10626,"thread_start_us":10439,"threads_started":1}
I20260927 15:14:28.818709 20974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 560011c1af4f4bdbb2da18b768863bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:14:28.822099 20840 tablet_service.cc:1467] Processing CreateTablet for tablet 560011c1af4f4bdbb2da18b768863bfc (DEFAULT_TABLE table=test-table [id=160550ef7fe84ac3a42373173eb9c7ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:14:28.823230 20840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 560011c1af4f4bdbb2da18b768863bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:14:28.833503 21192 tablet_bootstrap.cc:492] T 560011c1af4f4bdbb2da18b768863bfc P bbcead55eaa742d584f24d36afd99501: Bootstrap starting.
I20260927 15:14:28.835934 21193 tablet_bootstrap.cc:492] T 560011c1af4f4bdbb2da18b768863bfc P 14573af1c7f049918d56ae41cdd32f08: Bootstrap starting.
I20260927 15:14:28.838168 21194 tablet_bootstrap.cc:492] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706: Bootstrap starting.
I20260927 15:14:28.838699 21192 tablet_bootstrap.cc:654] T 560011c1af4f4bdbb2da18b768863bfc P bbcead55eaa742d584f24d36afd99501: Neither blocks nor log segments found. Creating new log.
I20260927 15:14:28.839365 21193 tablet_bootstrap.cc:654] T 560011c1af4f4bdbb2da18b768863bfc P 14573af1c7f049918d56ae41cdd32f08: Neither blocks nor log segments found. Creating new log.
I20260927 15:14:28.839715 21192 log.cc:824] T 560011c1af4f4bdbb2da18b768863bfc P bbcead55eaa742d584f24d36afd99501: Log is configured to *not* fsync() on all Append() calls
I20260927 15:14:28.840552 21193 log.cc:824] T 560011c1af4f4bdbb2da18b768863bfc P 14573af1c7f049918d56ae41cdd32f08: Log is configured to *not* fsync() on all Append() calls
I20260927 15:14:28.841406 21194 tablet_bootstrap.cc:654] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706: Neither blocks nor log segments found. Creating new log.
I20260927 15:14:28.842515 21192 tablet_bootstrap.cc:492] T 560011c1af4f4bdbb2da18b768863bfc P bbcead55eaa742d584f24d36afd99501: No bootstrap required, opened a new log
I20260927 15:14:28.842510 21194 log.cc:824] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706: Log is configured to *not* fsync() on all Append() calls
I20260927 15:14:28.842762 21192 ts_tablet_manager.cc:1397] T 560011c1af4f4bdbb2da18b768863bfc P bbcead55eaa742d584f24d36afd99501: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 15:14:28.843434 21193 tablet_bootstrap.cc:492] T 560011c1af4f4bdbb2da18b768863bfc P 14573af1c7f049918d56ae41cdd32f08: No bootstrap required, opened a new log
I20260927 15:14:28.843664 21193 ts_tablet_manager.cc:1397] T 560011c1af4f4bdbb2da18b768863bfc P 14573af1c7f049918d56ae41cdd32f08: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260927 15:14:28.845465 21194 tablet_bootstrap.cc:492] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706: No bootstrap required, opened a new log
I20260927 15:14:28.845746 21194 ts_tablet_manager.cc:1397] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 15:14:28.850740 21192 raft_consensus.cc:348] T 560011c1af4f4bdbb2da18b768863bfc P bbcead55eaa742d584f24d36afd99501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbcead55eaa742d584f24d36afd99501" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 41625 } } peers { permanent_uuid: "6840f5ea9ca2448b989c3d0a24767706" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33371 } } peers { permanent_uuid: "14573af1c7f049918d56ae41cdd32f08" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36411 } }
I20260927 15:14:28.851183 21192 raft_consensus.cc:374] T 560011c1af4f4bdbb2da18b768863bfc P bbcead55eaa742d584f24d36afd99501 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:14:28.851275 21192 raft_consensus.cc:729] T 560011c1af4f4bdbb2da18b768863bfc P bbcead55eaa742d584f24d36afd99501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbcead55eaa742d584f24d36afd99501, State: Initialized, Role: FOLLOWER
I20260927 15:14:28.851891 21192 consensus_queue.cc:260] T 560011c1af4f4bdbb2da18b768863bfc P bbcead55eaa742d584f24d36afd99501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbcead55eaa742d584f24d36afd99501" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 41625 } } peers { permanent_uuid: "6840f5ea9ca2448b989c3d0a24767706" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33371 } } peers { permanent_uuid: "14573af1c7f049918d56ae41cdd32f08" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36411 } }
I20260927 15:14:28.852973 21173 heartbeater.cc:503] Master 127.10.33.126:45061 was elected leader, sending a full tablet report...
I20260927 15:14:28.853978 21193 raft_consensus.cc:348] T 560011c1af4f4bdbb2da18b768863bfc P 14573af1c7f049918d56ae41cdd32f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbcead55eaa742d584f24d36afd99501" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 41625 } } peers { permanent_uuid: "6840f5ea9ca2448b989c3d0a24767706" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33371 } } peers { permanent_uuid: "14573af1c7f049918d56ae41cdd32f08" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36411 } }
I20260927 15:14:28.854446 21193 raft_consensus.cc:374] T 560011c1af4f4bdbb2da18b768863bfc P 14573af1c7f049918d56ae41cdd32f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:14:28.854539 21193 raft_consensus.cc:729] T 560011c1af4f4bdbb2da18b768863bfc P 14573af1c7f049918d56ae41cdd32f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14573af1c7f049918d56ae41cdd32f08, State: Initialized, Role: FOLLOWER
I20260927 15:14:28.855001 21192 ts_tablet_manager.cc:1428] T 560011c1af4f4bdbb2da18b768863bfc P bbcead55eaa742d584f24d36afd99501: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260927 15:14:28.855187 21193 consensus_queue.cc:260] T 560011c1af4f4bdbb2da18b768863bfc P 14573af1c7f049918d56ae41cdd32f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbcead55eaa742d584f24d36afd99501" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 41625 } } peers { permanent_uuid: "6840f5ea9ca2448b989c3d0a24767706" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33371 } } peers { permanent_uuid: "14573af1c7f049918d56ae41cdd32f08" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36411 } }
I20260927 15:14:28.856360 21039 heartbeater.cc:503] Master 127.10.33.126:45061 was elected leader, sending a full tablet report...
I20260927 15:14:28.857061 21193 ts_tablet_manager.cc:1428] T 560011c1af4f4bdbb2da18b768863bfc P 14573af1c7f049918d56ae41cdd32f08: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
I20260927 15:14:28.858467 21194 raft_consensus.cc:348] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbcead55eaa742d584f24d36afd99501" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 41625 } } peers { permanent_uuid: "6840f5ea9ca2448b989c3d0a24767706" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33371 } } peers { permanent_uuid: "14573af1c7f049918d56ae41cdd32f08" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36411 } }
I20260927 15:14:28.858935 21194 raft_consensus.cc:374] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:14:28.859045 21194 raft_consensus.cc:729] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6840f5ea9ca2448b989c3d0a24767706, State: Initialized, Role: FOLLOWER
I20260927 15:14:28.859675 21194 consensus_queue.cc:260] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbcead55eaa742d584f24d36afd99501" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 41625 } } peers { permanent_uuid: "6840f5ea9ca2448b989c3d0a24767706" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33371 } } peers { permanent_uuid: "14573af1c7f049918d56ae41cdd32f08" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36411 } }
I20260927 15:14:28.861433 21194 ts_tablet_manager.cc:1428] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
W20260927 15:14:28.878165 20906 tablet.cc:2369] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:14:28.925951 21040 tablet.cc:2369] T 560011c1af4f4bdbb2da18b768863bfc P 14573af1c7f049918d56ae41cdd32f08: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:14:28.967756 21174 tablet.cc:2369] T 560011c1af4f4bdbb2da18b768863bfc P bbcead55eaa742d584f24d36afd99501: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:14:29.080790 21200 raft_consensus.cc:482] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:14:29.081073 21200 raft_consensus.cc:504] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbcead55eaa742d584f24d36afd99501" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 41625 } } peers { permanent_uuid: "6840f5ea9ca2448b989c3d0a24767706" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33371 } } peers { permanent_uuid: "14573af1c7f049918d56ae41cdd32f08" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36411 } }
I20260927 15:14:29.082962 21200 leader_election.cc:288] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbcead55eaa742d584f24d36afd99501 (127.10.33.67:41625), 14573af1c7f049918d56ae41cdd32f08 (127.10.33.66:36411)
I20260927 15:14:29.090247 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:29.083440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:57611 (local address 127.10.33.66:36411)
0927 15:14:29.083651 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:29.083658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:29.084330 (+ 672us) server_negotiation.cc:407] Connection header received
0927 15:14:29.084504 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:29.084509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:29.084585 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:29.084668 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:14:29.085727 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:29.086516 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:29.088474 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:29.089236 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:29.089367 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:29.089449 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:29.089837 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 15:14:29.090000 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 15:14:29.090252 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:29.083380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:54905 (local address 127.10.33.67:41625)
0927 15:14:29.083586 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:29.083591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:29.084499 (+ 908us) server_negotiation.cc:407] Connection header received
0927 15:14:29.084811 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:29.084819 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:29.084914 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:29.085072 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:14:29.085755 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:29.086514 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:29.088450 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:29.089304 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:29.089480 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:29.089552 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:29.089883 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 15:14:29.090011 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:14:29.090463 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:29.083335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:36411 (local address 127.10.33.65:57611)
0927 15:14:29.084191 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:29.084216 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:29.084374 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:29.084811 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:29.084818 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:29.084870 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:14:29.085549 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:29.085562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:29.086647 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:29.086653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:29.088315 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:29.088327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:29.089468 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:29.089475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:29.089675 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:29.089709 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 15:14:29.089788 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 15:14:29.089965 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":57,"thread_start_us":444,"threads_started":1}
I20260927 15:14:29.090534 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:29.083552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:41625 (local address 127.10.33.65:54905)
0927 15:14:29.084381 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0927 15:14:29.084397 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:14:29.084525 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:14:29.085130 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:14:29.085134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:14:29.085158 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:14:29.085562 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:29.085571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:29.086652 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:29.086657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:29.088277 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:14:29.088290 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:14:29.089539 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:14:29.089543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:14:29.089699 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:29.089724 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:14:29.089810 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 15:14:29.089998 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":27,"thread_start_us":498,"threads_started":1}
I20260927 15:14:29.091454 20994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "560011c1af4f4bdbb2da18b768863bfc" candidate_uuid: "6840f5ea9ca2448b989c3d0a24767706" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14573af1c7f049918d56ae41cdd32f08" is_pre_election: true
I20260927 15:14:29.091642 21128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "560011c1af4f4bdbb2da18b768863bfc" candidate_uuid: "6840f5ea9ca2448b989c3d0a24767706" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbcead55eaa742d584f24d36afd99501" is_pre_election: true
I20260927 15:14:29.092099 20994 raft_consensus.cc:2454] T 560011c1af4f4bdbb2da18b768863bfc P 14573af1c7f049918d56ae41cdd32f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6840f5ea9ca2448b989c3d0a24767706 in term 0.
I20260927 15:14:29.092098 21128 raft_consensus.cc:2454] T 560011c1af4f4bdbb2da18b768863bfc P bbcead55eaa742d584f24d36afd99501 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6840f5ea9ca2448b989c3d0a24767706 in term 0.
I20260927 15:14:29.092844 20795 leader_election.cc:302] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14573af1c7f049918d56ae41cdd32f08, 6840f5ea9ca2448b989c3d0a24767706; no voters:
I20260927 15:14:29.093345 21200 raft_consensus.cc:2781] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:14:29.093542 21200 raft_consensus.cc:482] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:14:29.093654 21200 raft_consensus.cc:3037] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:14:29.096019 21200 raft_consensus.cc:504] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbcead55eaa742d584f24d36afd99501" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 41625 } } peers { permanent_uuid: "6840f5ea9ca2448b989c3d0a24767706" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33371 } } peers { permanent_uuid: "14573af1c7f049918d56ae41cdd32f08" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36411 } }
I20260927 15:14:29.096733 21200 leader_election.cc:288] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706 [CANDIDATE]: Term 1 election: Requested vote from peers bbcead55eaa742d584f24d36afd99501 (127.10.33.67:41625), 14573af1c7f049918d56ae41cdd32f08 (127.10.33.66:36411)
I20260927 15:14:29.097249 21128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "560011c1af4f4bdbb2da18b768863bfc" candidate_uuid: "6840f5ea9ca2448b989c3d0a24767706" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbcead55eaa742d584f24d36afd99501"
I20260927 15:14:29.097321 20994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "560011c1af4f4bdbb2da18b768863bfc" candidate_uuid: "6840f5ea9ca2448b989c3d0a24767706" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14573af1c7f049918d56ae41cdd32f08"
I20260927 15:14:29.097440 21128 raft_consensus.cc:3037] T 560011c1af4f4bdbb2da18b768863bfc P bbcead55eaa742d584f24d36afd99501 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:14:29.097587 20994 raft_consensus.cc:3037] T 560011c1af4f4bdbb2da18b768863bfc P 14573af1c7f049918d56ae41cdd32f08 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:14:29.099941 20994 raft_consensus.cc:2454] T 560011c1af4f4bdbb2da18b768863bfc P 14573af1c7f049918d56ae41cdd32f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6840f5ea9ca2448b989c3d0a24767706 in term 1.
I20260927 15:14:29.100036 21128 raft_consensus.cc:2454] T 560011c1af4f4bdbb2da18b768863bfc P bbcead55eaa742d584f24d36afd99501 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6840f5ea9ca2448b989c3d0a24767706 in term 1.
I20260927 15:14:29.100503 20795 leader_election.cc:302] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14573af1c7f049918d56ae41cdd32f08, 6840f5ea9ca2448b989c3d0a24767706; no voters:
I20260927 15:14:29.100857 21200 raft_consensus.cc:2781] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:14:29.101322 21200 raft_consensus.cc:686] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706 [term 1 LEADER]: Becoming Leader. State: Replica: 6840f5ea9ca2448b989c3d0a24767706, State: Running, Role: LEADER
I20260927 15:14:29.101907 21200 consensus_queue.cc:237] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbcead55eaa742d584f24d36afd99501" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 41625 } } peers { permanent_uuid: "6840f5ea9ca2448b989c3d0a24767706" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33371 } } peers { permanent_uuid: "14573af1c7f049918d56ae41cdd32f08" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36411 } }
I20260927 15:14:29.106300 20715 catalog_manager.cc:5495] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6840f5ea9ca2448b989c3d0a24767706 (127.10.33.65). New cstate: current_term: 1 leader_uuid: "6840f5ea9ca2448b989c3d0a24767706" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbcead55eaa742d584f24d36afd99501" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 41625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6840f5ea9ca2448b989c3d0a24767706" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14573af1c7f049918d56ae41cdd32f08" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36411 } health_report { overall_health: UNKNOWN } } }
I20260927 15:14:29.176951 21191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:29.171587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35488 (local address 127.10.33.65:33371)
0927 15:14:29.171799 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:29.171805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:29.171996 (+ 191us) server_negotiation.cc:407] Connection header received
0927 15:14:29.172204 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:29.172208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:29.172302 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:29.172436 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:14:29.172815 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:29.173592 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:29.174919 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:29.175321 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:29.175438 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:29.175907 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:14:29.176392 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:14:29.176524 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:29.176612 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 15:14:29.176692 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 15:14:29.185068 21128 raft_consensus.cc:1261] T 560011c1af4f4bdbb2da18b768863bfc P bbcead55eaa742d584f24d36afd99501 [term 1 FOLLOWER]: Refusing update from remote peer 6840f5ea9ca2448b989c3d0a24767706: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:14:29.185082 20994 raft_consensus.cc:1261] T 560011c1af4f4bdbb2da18b768863bfc P 14573af1c7f049918d56ae41cdd32f08 [term 1 FOLLOWER]: Refusing update from remote peer 6840f5ea9ca2448b989c3d0a24767706: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:14:29.186024 21203 consensus_queue.cc:1035] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbcead55eaa742d584f24d36afd99501" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:14:29.186398 21200 consensus_queue.cc:1035] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14573af1c7f049918d56ae41cdd32f08" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:14:29.223455 20716 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:14:29.223709 20716 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:14:29.227734 20716 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38ed0679a3484d48afeb662cb2e588bd: Sending DeleteTablet for 3 replicas of tablet 560011c1af4f4bdbb2da18b768863bfc
I20260927 15:14:29.229091 20974 tablet_service.cc:1514] Processing DeleteTablet for tablet 560011c1af4f4bdbb2da18b768863bfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:14:29 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:50422
I20260927 15:14:29.229403 21108 tablet_service.cc:1514] Processing DeleteTablet for tablet 560011c1af4f4bdbb2da18b768863bfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:14:29 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:54812
I20260927 15:14:29.229763 20840 tablet_service.cc:1514] Processing DeleteTablet for tablet 560011c1af4f4bdbb2da18b768863bfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:14:29 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:35478
I20260927 15:14:29.231400 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 20777
I20260927 15:14:29.241729 21217 tablet_replica.cc:330] T 560011c1af4f4bdbb2da18b768863bfc P bbcead55eaa742d584f24d36afd99501: stopping tablet replica
I20260927 15:14:29.242333 21217 raft_consensus.cc:2229] T 560011c1af4f4bdbb2da18b768863bfc P bbcead55eaa742d584f24d36afd99501 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:14:29.242926 21217 raft_consensus.cc:2258] T 560011c1af4f4bdbb2da18b768863bfc P bbcead55eaa742d584f24d36afd99501 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:14:29.243803 21216 tablet_replica.cc:330] T 560011c1af4f4bdbb2da18b768863bfc P 14573af1c7f049918d56ae41cdd32f08: stopping tablet replica
I20260927 15:14:29.244230 21216 raft_consensus.cc:2229] T 560011c1af4f4bdbb2da18b768863bfc P 14573af1c7f049918d56ae41cdd32f08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:14:29.244736 21218 tablet_replica.cc:330] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706: stopping tablet replica
I20260927 15:14:29.244735 21216 raft_consensus.cc:2258] T 560011c1af4f4bdbb2da18b768863bfc P 14573af1c7f049918d56ae41cdd32f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:14:29.245133 21218 raft_consensus.cc:2229] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:14:29.245918 21218 raft_consensus.cc:2258] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:14:29.247097 21216 ts_tablet_manager.cc:1905] T 560011c1af4f4bdbb2da18b768863bfc P 14573af1c7f049918d56ae41cdd32f08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:14:29.247946 21217 ts_tablet_manager.cc:1905] T 560011c1af4f4bdbb2da18b768863bfc P bbcead55eaa742d584f24d36afd99501: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:14:29.248884 21218 ts_tablet_manager.cc:1905] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:14:29.254340 21216 ts_tablet_manager.cc:1918] T 560011c1af4f4bdbb2da18b768863bfc P 14573af1c7f049918d56ae41cdd32f08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:14:29.254383 21217 ts_tablet_manager.cc:1918] T 560011c1af4f4bdbb2da18b768863bfc P bbcead55eaa742d584f24d36afd99501: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:14:29.254526 21217 log.cc:1196] T 560011c1af4f4bdbb2da18b768863bfc P bbcead55eaa742d584f24d36afd99501: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/560011c1af4f4bdbb2da18b768863bfc
I20260927 15:14:29.254527 21216 log.cc:1196] T 560011c1af4f4bdbb2da18b768863bfc P 14573af1c7f049918d56ae41cdd32f08: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/560011c1af4f4bdbb2da18b768863bfc
I20260927 15:14:29.255014 21216 ts_tablet_manager.cc:1939] T 560011c1af4f4bdbb2da18b768863bfc P 14573af1c7f049918d56ae41cdd32f08: Deleting consensus metadata
I20260927 15:14:29.255014 21217 ts_tablet_manager.cc:1939] T 560011c1af4f4bdbb2da18b768863bfc P bbcead55eaa742d584f24d36afd99501: Deleting consensus metadata
I20260927 15:14:29.256964 20703 catalog_manager.cc:4841] TS bbcead55eaa742d584f24d36afd99501 (127.10.33.67:41625): tablet 560011c1af4f4bdbb2da18b768863bfc (table test-table [id=160550ef7fe84ac3a42373173eb9c7ef]) successfully deleted
I20260927 15:14:29.257074 20702 catalog_manager.cc:4841] TS 14573af1c7f049918d56ae41cdd32f08 (127.10.33.66:36411): tablet 560011c1af4f4bdbb2da18b768863bfc (table test-table [id=160550ef7fe84ac3a42373173eb9c7ef]) successfully deleted
Sep 27 15:14:29 dist-test-slave-f5h6 krb5kdc[20671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.65@KRBTEST.COM
I20260927 15:14:29.262005 21218 ts_tablet_manager.cc:1918] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:14:29.262187 21218 log.cc:1196] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/560011c1af4f4bdbb2da18b768863bfc
I20260927 15:14:29.262698 21218 ts_tablet_manager.cc:1939] T 560011c1af4f4bdbb2da18b768863bfc P 6840f5ea9ca2448b989c3d0a24767706: Deleting consensus metadata
I20260927 15:14:29.264376 20703 catalog_manager.cc:4841] TS 6840f5ea9ca2448b989c3d0a24767706 (127.10.33.65:33371): tablet 560011c1af4f4bdbb2da18b768863bfc (table test-table [id=160550ef7fe84ac3a42373173eb9c7ef]) successfully deleted
I20260927 15:14:29.269186 21191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:29.233074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35500 (local address 127.10.33.65:33371)
0927 15:14:29.233253 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:29.233259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:29.244072 (+ 10813us) server_negotiation.cc:407] Connection header received
0927 15:14:29.244291 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:29.244297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:29.244458 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:29.244581 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:29.249165 (+ 4584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:29.250410 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:29.254672 (+ 4262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:29.255397 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:29.255518 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:29.261656 (+ 6138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:29.261673 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:29.261693 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:14:29.261747 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:29.266015 (+ 4268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:29.266805 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:29.266815 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:29.266818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:29.267027 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:29.267479 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:29.267484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:29.267487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:29.267969 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:29.268281 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:29.268556 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 15:14:29.268810 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:14:29.270020 20900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:14:29.371850 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 20777
I20260927 15:14:29.386371 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 20911
Sep 27 15:14:29 dist-test-slave-f5h6 krb5kdc[20671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.66@KRBTEST.COM
I20260927 15:14:29.400365 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:29.387071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50430 (local address 127.10.33.66:36411)
0927 15:14:29.387252 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:29.387258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:29.387278 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:14:29.387416 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:29.387422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:29.387489 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:29.387571 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:29.388715 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:29.389420 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:29.390681 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:29.391074 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:29.391182 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:29.395203 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:29.395216 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:29.395232 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:14:29.395271 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:29.397696 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:29.398352 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:29.398361 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:29.398364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:29.398494 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:29.398854 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:29.398857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:29.398859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:29.399230 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:29.399509 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:29.399806 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 15:14:29.400028 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:14:29.401098 21034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:14:29.489344 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 20911
I20260927 15:14:29.503155 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 21045
Sep 27 15:14:29 dist-test-slave-f5h6 krb5kdc[20671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.67@KRBTEST.COM
I20260927 15:14:29.518741 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:14:29.503926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54814 (local address 127.10.33.67:41625)
0927 15:14:29.504078 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 15:14:29.504082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:14:29.504135 (+ 53us) server_negotiation.cc:407] Connection header received
0927 15:14:29.504307 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:14:29.504311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:14:29.504376 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:14:29.504522 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:14:29.505752 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:29.506528 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:29.507897 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:14:29.508281 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:14:29.508400 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:14:29.512687 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:14:29.512704 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:14:29.512722 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:14:29.512764 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:14:29.515872 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:29.516623 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:29.516631 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:29.516633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:29.516784 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:14:29.517170 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:14:29.517175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:14:29.517177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:14:29.517619 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:14:29.517973 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0927 15:14:29.518315 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 15:14:29.518490 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:14:29.519488 21168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:14:29.609048 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 21045
I20260927 15:14:29.622470 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 20684
I20260927 15:14:29.623626 20746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:14:29.729830 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 20684
2026-09-27T15:14:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4485 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:14:29 dist-test-slave-f5h6 krb5kdc[21226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:14:29 dist-test-slave-f5h6 krb5kdc[21226](info): set up 2 sockets
Sep 27 15:14:29 dist-test-slave-f5h6 krb5kdc[21226](info): commencing operation
krb5kdc: starting...
W20260927 15:14:30.762631 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:14:30 dist-test-slave-f5h6 krb5kdc[21226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:14:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:14:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.33.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.33.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.33.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/rangeradmin_127.10.33.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.33.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/rangeradmin_127.10.33.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.33.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.33.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.33.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/rangerlookup_127.10.33.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.33.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/rangerlookup_127.10.33.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.33.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.33.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.33.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/HTTP_127.10.33.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.33.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/HTTP_127.10.33.84@KRBTEST.COM.keytab.
I20260927 15:14:31.098547 10373 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskQegxT3/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/postgres -l logfile start
2026-09-27 15:14:33.871 UTC [21256] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 15:14:33.871 UTC [21256] LOG: listening on IPv4 address "127.10.33.84", port 35435
2026-09-27 15:14:33.875 UTC [21256] LOG: listening on Unix socket "/tmp/.s.PGSQL.35435"
2026-09-27 15:14:33.898 UTC [21259] LOG: database system was shut down at 2026-09-27 15:14:32 UTC
2026-09-27 15:14:33.902 UTC [21256] LOG: database system is ready to accept connections
I20260927 15:14:34.802376 10373 mini_postgres.cc:96] Postgres bound to 35435
2026-09-27 15:14:34.819 UTC [21267] FATAL: database "slave" does not exist
127.10.33.84:35435 - accepting connections
I20260927 15:14:34.819782 10373 mini_ranger.cc:161] Starting Ranger...
I20260927 15:14:34.845345 10373 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 15:14:35.432442 10373 mini_ranger.cc:87] Created ranger Postgres database
I20260927 15:14:35.432586 10373 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ranger-admin
2026-09-27 15:14:35,732 [I] DB FLAVOR :POSTGRES
2026-09-27 15:14:35,733 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:14:35,733 [I] Checking connection..
2026-09-27 15:14:35,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 15:14:36,000 [I] Checking connection passed.
2026-09-27 15:14:36,000 [I] --------- Verifying version history table ---------
2026-09-27 15:14:36,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:14:36,248 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:14:36,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:14:36,527 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:14:36,527 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 15:14:36,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 15:14:36.794 UTC [21331] WARNING: there is no transaction in progress
2026-09-27 15:14:36,801 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 15:14:36,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:14:37,058 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 15:14:37,059 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:14:37,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 15:14:37,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 15:14:37,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f5h6.c.gcp-upstream.internal','N') ;"
2026-09-27 15:14:37,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:14:38,098 [I] Table x_portal_user does not exist in database ranger
2026-09-27 15:14:38,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:14:38,358 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 15:14:38,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:14:38,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:14:38,873 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 15:14:38,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 15:14:39.953 UTC [21439] WARNING: there is no transaction in progress
2026-09-27 15:14:39.974 UTC [21439] WARNING: there is no transaction in progress
2026-09-27 15:14:39.999 UTC [21439] WARNING: there is no transaction in progress
2026-09-27 15:14:40.018 UTC [21439] WARNING: there is no transaction in progress
2026-09-27 15:14:40.037 UTC [21439] WARNING: there is no transaction in progress
2026-09-27 15:14:40.059 UTC [21439] WARNING: there is no transaction in progress
2026-09-27 15:14:40.127 UTC [21439] WARNING: there is no transaction in progress
2026-09-27 15:14:40.135 UTC [21439] WARNING: there is no transaction in progress
2026-09-27 15:14:40.153 UTC [21439] WARNING: there is no transaction in progress
2026-09-27 15:14:40.166 UTC [21439] WARNING: there is no transaction in progress
2026-09-27 15:14:40.178 UTC [21439] WARNING: there is no transaction in progress
2026-09-27 15:14:40.195 UTC [21439] WARNING: there is no transaction in progress
2026-09-27 15:14:40.209 UTC [21439] WARNING: there is no transaction in progress
2026-09-27 15:14:40.218 UTC [21439] WARNING: there is no transaction in progress
2026-09-27 15:14:40.228 UTC [21439] WARNING: there is no transaction in progress
2026-09-27 15:14:40.238 UTC [21439] WARNING: there is no transaction in progress
2026-09-27 15:14:41,088 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 15:14:41,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 15:14:41,354 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 15:14:41,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:14:41,630 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 15:14:41,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:14:41,878 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 15:14:41,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:14:42,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:14:42,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f5h6.c.gcp-upstream.internal';"
2026-09-27 15:14:42,662 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 15:14:42,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 15:14:42,928 [I] DB_PATCHES have already been applied
I20260927 15:14:42.934023 10373 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ranger-admin:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskQegxT3/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 15:14:42.934144 10373 mini_ranger.cc:193] Using host: 127.10.33.84
I20260927 15:14:42.948388 10373 mini_ranger.cc:227] Ranger admin URL: http://127.10.33.84:35769
Sep 27, 2026 3:14:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:14:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:14:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 3:14:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 3:14:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 3:14:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 3:14:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 15:14:44.008710 10373 mini_ranger.cc:160] Time spent starting Ranger: real 9.189s user 0.002s sys 0.039s
Sep 27, 2026 3:15:04 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 3:15:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 3:15:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 3:15:04 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 3:15:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 3:15:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 3:15:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 3:15:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 3:15:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 3:15:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 3:15:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 3:15:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 3:15:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 3:15:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 3:15:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 3:15:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 3:15:04 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 3:15:05 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 15:15:05.963954 10373 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.33.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.33.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.33.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/rangerkms_127.10.33.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.33.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/rangerkms_127.10.33.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.33.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.33.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.33.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/HTTP_127.10.33.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.33.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/HTTP_127.10.33.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 15:15:06 dist-test-slave-f5h6 krb5kdc[21226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522106, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 15:15:06.134919 10373 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 15:15:06.154798 10373 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 15:15:07.186785 10373 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 15:15:07.186929 10373 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ranger-kms
I20260927 15:15:07.187170 10373 mini_ranger_kms.cc:227] Using postgres at 127.10.33.84:35435
2026-09-27 15:15:07,595 [I] DB FLAVOR :POSTGRES
2026-09-27 15:15:07,596 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:15:07,596 [I] Checking connection
2026-09-27 15:15:07,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 15:15:07,838 [I] connection success
2026-09-27 15:15:07,839 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 15:15:07,839 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 15:15:07,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 15:15:08,083 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 15:15:08,084 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:15:08,084 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 15:15:08,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:35435/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 15:15:08,357 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 15:15:08.818436 10373 mini_ranger_kms.cc:310] Created kms service
I20260927 15:15:08.999495 10373 mini_ranger_kms.cc:326] Created kudu user
I20260927 15:15:09.041363 10373 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 15:15:09 dist-test-slave-f5h6 krb5kdc[21226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522106, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
I20260927 15:15:09.486582 10373 mini_ranger_kms.cc:384] Added ranger policy
I20260927 15:15:09.486860 10373 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ranger-kms:/tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 15:15:09.486904 10373 mini_ranger_kms.cc:240] Using host: 127.10.33.84
I20260927 15:15:09.504768 10373 mini_ranger_kms.cc:276] Ranger KMS PID: 21620
I20260927 15:15:09.504978 10373 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.33.84:39239
Sep 27, 2026 3:15:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:15:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:15:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 3:15:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 3:15:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 3:15:10 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 15:15:11.591368 10373 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.456s user 0.131s sys 0.303s
I20260927 15:15:11.591646 10373 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 15:15:11.591863 10373 mini_ranger_kms.cc:401] 127.10.33.84:39239/kms/v1/keys
Sep 27 15:15:11 dist-test-slave-f5h6 krb5kdc[21226](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790522111, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.33.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 15:15:12 dist-test-slave-f5h6 krb5kdc[21226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522111, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.33.84@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
Sep 27 15:15:12 dist-test-slave-f5h6 krb5kdc[21226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522111, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.33.84@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
Sep 27 15:15:14 dist-test-slave-f5h6 krb5kdc[21226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:15:14.761622 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:34031
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:34031
--ranger_config_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.33.84:39239/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:15:15.087265 21670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:15:15.087572 21670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:15:15.087687 21670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:15:15.095031 21670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:15:15.095151 21670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:15:15.095203 21670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:15:15.095281 21670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:15:15.095352 21670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:15:15.105824 21670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.33.84:39239/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:34031
--ranger_config_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:34031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:15:15.107189 21670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:15:15.108482 21670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:15:15.120573 21675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:15:15.120520 21676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:15:15.120488 21678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:15:15.121542 21670 server_base.cc:1034] running on GCE node
I20260927 15:15:15.122380 21670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:15:15.123811 21670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:15:15.125022 21670 hybrid_clock.cc:648] HybridClock initialized: now 1790522115124983 us; error 31 us; skew 500 ppm
I20260927 15:15:15.125422 21670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:15:15 dist-test-slave-f5h6 krb5kdc[21226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522115, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:15:15.137739 21670 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:15:15.139333 21670 webserver.cc:460] Webserver started at http://127.10.33.126:45681/ using document root <none> and password file <none>
I20260927 15:15:15.139990 21670 fs_manager.cc:359] Metadata directory not provided
I20260927 15:15:15.140095 21670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:15:15.140411 21670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:15:15 dist-test-slave-f5h6 krb5kdc[21226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522115, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
I20260927 15:15:15.198210 21670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "94295f3a55f348c1be8f2a0c224d618e"
format_stamp: "Formatted at 2026-09-27 15:15:15 on dist-test-slave-f5h6"
server_key: "179a8029beb497ded6f34393d518489f"
server_key_iv: "0fc0f59ef228ee4ae7491b5987e034ee"
server_key_version: "kuduclusterkey@0"
I20260927 15:15:15.199055 21670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "94295f3a55f348c1be8f2a0c224d618e"
format_stamp: "Formatted at 2026-09-27 15:15:15 on dist-test-slave-f5h6"
server_key: "179a8029beb497ded6f34393d518489f"
server_key_iv: "0fc0f59ef228ee4ae7491b5987e034ee"
server_key_version: "kuduclusterkey@0"
I20260927 15:15:15.204454 21670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 15:15:15.224889 21685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:15:15.226780 21670 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 15:15:15.227010 21670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "94295f3a55f348c1be8f2a0c224d618e"
format_stamp: "Formatted at 2026-09-27 15:15:15 on dist-test-slave-f5h6"
server_key: "179a8029beb497ded6f34393d518489f"
server_key_iv: "0fc0f59ef228ee4ae7491b5987e034ee"
server_key_version: "kuduclusterkey@0"
I20260927 15:15:15.227294 21670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:15:15.270454 21670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:15:15.274607 21670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:15:15.274878 21670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:15:15.292753 21670 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:34031
I20260927 15:15:15.292806 21736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:34031 every 8 connection(s)
I20260927 15:15:15.294157 21670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:15:15.298144 21737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:15:15.298840 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 21670
I20260927 15:15:15.299041 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:15:15.299896 10373 external_mini_cluster.cc:1428] Setting key 3db0aa03949ebdf4fcd969b9ff3262b5
I20260927 15:15:15.312712 21737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e: Bootstrap starting.
Sep 27 15:15:15 dist-test-slave-f5h6 krb5kdc[21226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:15:15.316422 21737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e: Neither blocks nor log segments found. Creating new log.
I20260927 15:15:15.317489 21737 log.cc:824] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e: Log is configured to *not* fsync() on all Append() calls
I20260927 15:15:15.320369 21737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e: No bootstrap required, opened a new log
I20260927 15:15:15.322980 21740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:15:15.302129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42872 (local address 127.10.33.126:34031)
0927 15:15:15.302952 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0927 15:15:15.302964 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:15:15.303014 (+ 50us) server_negotiation.cc:407] Connection header received
0927 15:15:15.304762 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:15:15.304795 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:15:15.305280 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:15:15.305658 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:15:15.307298 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:15.308611 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:15.310401 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:15.311199 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:15.311361 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:15:15.315898 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:15:15.315907 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:15:15.315923 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:15:15.315975 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:15:15.318991 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:15:15.319714 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:15:15.319721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:15:15.319727 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:15:15.319870 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:15:15.320229 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:15:15.320233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:15:15.320235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:15:15.320807 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:15:15.321087 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 15:15:15.321543 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 15:15:15.321972 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":362,"threads_started":1}
I20260927 15:15:15.328356 21737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94295f3a55f348c1be8f2a0c224d618e" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 34031 } }
I20260927 15:15:15.328712 21737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:15:15.328804 21737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94295f3a55f348c1be8f2a0c224d618e, State: Initialized, Role: FOLLOWER
I20260927 15:15:15.329335 21737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94295f3a55f348c1be8f2a0c224d618e" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 34031 } }
I20260927 15:15:15.329548 21737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:15:15.329668 21737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:15:15.329803 21737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:15:15.332513 21737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94295f3a55f348c1be8f2a0c224d618e" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 34031 } }
I20260927 15:15:15.333081 21737 leader_election.cc:302] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94295f3a55f348c1be8f2a0c224d618e; no voters:
I20260927 15:15:15.333690 21737 leader_election.cc:288] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:15:15.333811 21742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:15:15.334472 21742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e [term 1 LEADER]: Becoming Leader. State: Replica: 94295f3a55f348c1be8f2a0c224d618e, State: Running, Role: LEADER
I20260927 15:15:15.335013 21742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94295f3a55f348c1be8f2a0c224d618e" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 34031 } }
I20260927 15:15:15.336019 21737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:15:15.337883 21744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94295f3a55f348c1be8f2a0c224d618e. Latest consensus state: current_term: 1 leader_uuid: "94295f3a55f348c1be8f2a0c224d618e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94295f3a55f348c1be8f2a0c224d618e" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 34031 } } }
I20260927 15:15:15.338325 21744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e [sys.catalog]: This master's current role is: LEADER
I20260927 15:15:15.337831 21743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94295f3a55f348c1be8f2a0c224d618e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94295f3a55f348c1be8f2a0c224d618e" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 34031 } } }
I20260927 15:15:15.338740 21743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e [sys.catalog]: This master's current role is: LEADER
I20260927 15:15:15.339885 21750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:15:15.344834 21750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:15:15.350111 21737 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 15:15:15.361222 21750 catalog_manager.cc:1347] Generated new cluster ID: 759f19ea68cf497584aa908a3e56a400
I20260927 15:15:15.361380 21750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:15:15.382290 21750 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:15:15.384532 21750 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:15:15.396029 21750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e: Generated new TSK 0
I20260927 15:15:15.397440 21750 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.33.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
I20260927 15:15:16.138514 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.33.84:39239/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:34031
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:15:16 dist-test-slave-f5h6 krb5kdc[21226](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790522116, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 15:15:16.535370 21782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:15:16.535669 21782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:15:16.535797 21782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:15:16.547103 21782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:15:16.547266 21782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:15:16.547523 21782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:15:16.563218 21782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.33.84:39239/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:34031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:15:16.565078 21782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:15:16.566861 21782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:15:16.592566 21792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:15:16.596642 21793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:15:16.597501 21782 server_base.cc:1034] running on GCE node
W20260927 15:15:16.597589 21796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:15:16.598387 21782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:15:16.599418 21782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:15:16.600669 21782 hybrid_clock.cc:648] HybridClock initialized: now 1790522116600607 us; error 54 us; skew 500 ppm
I20260927 15:15:16.601007 21782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:15:16 dist-test-slave-f5h6 krb5kdc[21226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522116, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:15:16.604122 21782 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:15:16.605954 21782 webserver.cc:460] Webserver started at http://127.10.33.65:45987/ using document root <none> and password file <none>
I20260927 15:15:16.606813 21782 fs_manager.cc:359] Metadata directory not provided
I20260927 15:15:16.606941 21782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:15:16.607260 21782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:15:16 dist-test-slave-f5h6 krb5kdc[21226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522116, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
I20260927 15:15:16.629948 21782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "b31c3b6911854dfcbbfbb714e728f388"
format_stamp: "Formatted at 2026-09-27 15:15:16 on dist-test-slave-f5h6"
server_key: "c788421ba83b035b36424133dd9ee60e"
server_key_iv: "323d701a669077c53e9acfa1aafbe19e"
server_key_version: "kuduclusterkey@0"
I20260927 15:15:16.630887 21782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "b31c3b6911854dfcbbfbb714e728f388"
format_stamp: "Formatted at 2026-09-27 15:15:16 on dist-test-slave-f5h6"
server_key: "c788421ba83b035b36424133dd9ee60e"
server_key_iv: "323d701a669077c53e9acfa1aafbe19e"
server_key_version: "kuduclusterkey@0"
I20260927 15:15:16.636929 21782 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 15:15:16.662231 21802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:15:16.663834 21782 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.005s sys 0.002s
I20260927 15:15:16.664122 21782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "b31c3b6911854dfcbbfbb714e728f388"
format_stamp: "Formatted at 2026-09-27 15:15:16 on dist-test-slave-f5h6"
server_key: "c788421ba83b035b36424133dd9ee60e"
server_key_iv: "323d701a669077c53e9acfa1aafbe19e"
server_key_version: "kuduclusterkey@0"
I20260927 15:15:16.664556 21782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:15:16.692588 21782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:15:16.697574 21782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:15:16.697943 21782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:15:16.699085 21782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:15:16.701224 21782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:15:16.701314 21782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:15:16.701434 21782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:15:16.701488 21782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:15:16.752429 21782 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:33257
I20260927 15:15:16.752445 21914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:33257 every 8 connection(s)
I20260927 15:15:16.754243 21782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:15:16.760970 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 21782
I20260927 15:15:16.761184 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:15:16.762044 10373 external_mini_cluster.cc:1428] Setting key eda26831821129711c686b19f7b4cc24
Sep 27 15:15:16 dist-test-slave-f5h6 krb5kdc[21226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522116, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.33.66@KRBTEST.COM; defaulting to no policy
I20260927 15:15:16.797721 21917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:15:16.757694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:50823 (local address 127.10.33.126:34031)
0927 15:15:16.777888 (+ 20194us) server_negotiation.cc:206] Beginning negotiation
0927 15:15:16.777896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:15:16.777919 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:15:16.777988 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:15:16.777994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:15:16.778090 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:15:16.778205 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:15:16.780689 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:16.781953 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:16.784355 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:16.785002 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:16.785132 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:15:16.789804 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:15:16.789813 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:15:16.789817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:15:16.789898 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:15:16.793618 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:15:16.794851 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:15:16.794858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:15:16.794862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:15:16.795012 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:15:16.795610 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:15:16.795614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:15:16.795616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:15:16.795907 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:15:16.796042 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 15:15:16.796946 (+ 904us) server_negotiation.cc:299] Negotiation successful
0927 15:15:16.797205 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20023,"thread_start_us":20615,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.66@KRBTEST.COM" created.
I20260927 15:15:16.799278 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:15:16.757977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:34031 (local address 127.10.33.65:50823)
0927 15:15:16.771816 (+ 13839us) negotiation.cc:107] Waiting for socket to connect
0927 15:15:16.771881 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 15:15:16.774132 (+ 2251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:15:16.778825 (+ 4693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:15:16.778840 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:15:16.779454 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:15:16.780430 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:15:16.780462 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:15:16.782183 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:15:16.782190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:15:16.784127 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:15:16.784146 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:15:16.785478 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:15:16.785485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:15:16.785795 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:15:16.786719 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:15:16.786750 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:15:16.789458 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:15:16.793943 (+ 4485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:15:16.793955 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:15:16.793976 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:15:16.794667 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:15:16.795157 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:15:16.795162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:15:16.795166 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:15:16.795456 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:15:16.796065 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:15:16.796075 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:15:16.796437 (+ 362us) client_negotiation.cc:769] Sending connection context
0927 15:15:16.796811 (+ 374us) client_negotiation.cc:240] Negotiation successful
0927 15:15:16.797553 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13500,"thread_start_us":13237,"threads_started":1}
I20260927 15:15:16.801628 21915 heartbeater.cc:347] Connected to a master server at 127.10.33.126:34031
I20260927 15:15:16.801993 21915 heartbeater.cc:464] Registering TS with master...
I20260927 15:15:16.802865 21915 heartbeater.cc:511] Master 127.10.33.126:34031 requested a full tablet report, sending...
I20260927 15:15:16.805450 21702 ts_manager.cc:194] Registered new tserver with Master: b31c3b6911854dfcbbfbb714e728f388 (127.10.33.65:33257)
I20260927 15:15:16.807471 21702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:50823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
I20260927 15:15:16.833961 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.33.84:39239/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:34031
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:15:16 dist-test-slave-f5h6 krb5kdc[21226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522116, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
Sep 27 15:15:17 dist-test-slave-f5h6 krb5kdc[21226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522116, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
W20260927 15:15:17.265404 21921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:15:17.265738 21921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:15:17.265918 21921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:15:17.274529 21921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:15:17.274662 21921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:15:17.274844 21921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:15:17.287092 21921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.33.84:39239/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:34031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:15:17.288758 21921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:15:17.290408 21921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:15:17.319713 21928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:15:17.321813 21929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:15:17.331920 21921 server_base.cc:1034] running on GCE node
W20260927 15:15:17.334139 21931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:15:17.335093 21921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:15:17.336858 21921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:15:17.339095 21921 hybrid_clock.cc:648] HybridClock initialized: now 1790522117339022 us; error 61 us; skew 500 ppm
I20260927 15:15:17.339541 21921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:15:17 dist-test-slave-f5h6 krb5kdc[21226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522117, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:15:17.344048 21921 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:15:17.365803 21921 webserver.cc:460] Webserver started at http://127.10.33.66:45389/ using document root <none> and password file <none>
I20260927 15:15:17.367014 21921 fs_manager.cc:359] Metadata directory not provided
I20260927 15:15:17.367249 21921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:15:17.367671 21921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:15:17 dist-test-slave-f5h6 krb5kdc[21226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522117, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
I20260927 15:15:17.396384 21921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "f23df7cb3cc94e0ca160fc2d3c0818d8"
format_stamp: "Formatted at 2026-09-27 15:15:17 on dist-test-slave-f5h6"
server_key: "cf43a3c39f20aeb4d664c6e963dd670c"
server_key_iv: "1a9c60b68b0559122047456740df3560"
server_key_version: "kuduclusterkey@0"
I20260927 15:15:17.397585 21921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "f23df7cb3cc94e0ca160fc2d3c0818d8"
format_stamp: "Formatted at 2026-09-27 15:15:17 on dist-test-slave-f5h6"
server_key: "cf43a3c39f20aeb4d664c6e963dd670c"
server_key_iv: "1a9c60b68b0559122047456740df3560"
server_key_version: "kuduclusterkey@0"
I20260927 15:15:17.418838 21921 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.007s sys 0.009s
I20260927 15:15:17.460922 21938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:15:17.463188 21921 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.009s sys 0.001s
I20260927 15:15:17.463496 21921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "f23df7cb3cc94e0ca160fc2d3c0818d8"
format_stamp: "Formatted at 2026-09-27 15:15:17 on dist-test-slave-f5h6"
server_key: "cf43a3c39f20aeb4d664c6e963dd670c"
server_key_iv: "1a9c60b68b0559122047456740df3560"
server_key_version: "kuduclusterkey@0"
I20260927 15:15:17.463886 21921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:15:17.520171 21921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:15:17.524747 21921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:15:17.525238 21921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:15:17.526912 21921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:15:17.528730 21921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:15:17.528868 21921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 15:15:17.529021 21921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:15:17.529107 21921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:15:17.562101 21921 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:35481
I20260927 15:15:17.562101 22057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:35481 every 8 connection(s)
I20260927 15:15:17.563380 21921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:15:17.566162 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 21921
I20260927 15:15:17.566347 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:15:17.566788 10373 external_mini_cluster.cc:1428] Setting key e56989e9b50a849efc4eecc349f74d26
Sep 27 15:15:17 dist-test-slave-f5h6 krb5kdc[21226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522117, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:15:17.585151 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:15:17.565836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:45877 (local address 127.10.33.126:34031)
0927 15:15:17.566361 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0927 15:15:17.566368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:15:17.567443 (+ 1075us) server_negotiation.cc:407] Connection header received
0927 15:15:17.569564 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:15:17.569570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:15:17.569661 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:15:17.569774 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:15:17.571539 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:17.572298 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:17.574442 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:17.574852 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:17.574945 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:15:17.578620 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:15:17.578626 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:15:17.578631 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:15:17.578669 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:15:17.581760 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:15:17.582589 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:15:17.582595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:15:17.582600 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:15:17.582752 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:15:17.583284 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:15:17.583289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:15:17.583292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:15:17.583650 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:15:17.583791 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 15:15:17.584548 (+ 757us) server_negotiation.cc:299] Negotiation successful
0927 15:15:17.584776 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":214,"threads_started":1}
I20260927 15:15:17.585976 22061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:15:17.566493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:34031 (local address 127.10.33.66:45877)
0927 15:15:17.567271 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0927 15:15:17.567323 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:15:17.569258 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:15:17.570041 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:15:17.570052 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:15:17.570613 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:15:17.571366 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:15:17.571388 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:15:17.572437 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:15:17.572443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:15:17.574302 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:15:17.574317 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:15:17.575062 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:15:17.575068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:15:17.575320 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:15:17.575982 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:15:17.576009 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:15:17.578364 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:15:17.581958 (+ 3594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:15:17.581966 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:15:17.581981 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:15:17.582455 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:15:17.582904 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:15:17.582907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:15:17.582910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:15:17.583155 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:15:17.583798 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:15:17.583805 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:15:17.584153 (+ 348us) client_negotiation.cc:769] Sending connection context
0927 15:15:17.584415 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 15:15:17.584813 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":312,"threads_started":1}
I20260927 15:15:17.587544 22058 heartbeater.cc:347] Connected to a master server at 127.10.33.126:34031
I20260927 15:15:17.587888 22058 heartbeater.cc:464] Registering TS with master...
I20260927 15:15:17.588744 22058 heartbeater.cc:511] Master 127.10.33.126:34031 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.33.67@KRBTEST.COM; defaulting to no policy
I20260927 15:15:17.590531 21702 ts_manager.cc:194] Registered new tserver with Master: f23df7cb3cc94e0ca160fc2d3c0818d8 (127.10.33.66:35481)
I20260927 15:15:17.591341 21702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:45877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
I20260927 15:15:17.621011 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.33.84:39239/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:34031
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:15:17.811524 21915 heartbeater.cc:503] Master 127.10.33.126:34031 was elected leader, sending a full tablet report...
W20260927 15:15:17.937464 22064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:15:17.937753 22064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:15:17.937927 22064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:15:17.945159 22064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:15:17.945293 22064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:15:17.945439 22064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:15:17.955317 22064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.33.84:39239/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:34031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:15:17.956615 22064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:15:17.957883 22064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:15:17.970829 22070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:15:17.970795 22072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:15:17.971009 22069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:15:17.971881 22064 server_base.cc:1034] running on GCE node
I20260927 15:15:17.972473 22064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:15:17.973486 22064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:15:17.974686 22064 hybrid_clock.cc:648] HybridClock initialized: now 1790522117974644 us; error 44 us; skew 500 ppm
I20260927 15:15:17.975031 22064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:15:17 dist-test-slave-f5h6 krb5kdc[21226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522117, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:15:17.978158 22064 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:15:17.979638 22064 webserver.cc:460] Webserver started at http://127.10.33.67:43163/ using document root <none> and password file <none>
I20260927 15:15:17.980302 22064 fs_manager.cc:359] Metadata directory not provided
I20260927 15:15:17.980415 22064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:15:17.980690 22064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:15:17 dist-test-slave-f5h6 krb5kdc[21226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522117, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
I20260927 15:15:18.004968 22064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "cdc0e9fba29c4ec1b58fab89b5b0155a"
format_stamp: "Formatted at 2026-09-27 15:15:18 on dist-test-slave-f5h6"
server_key: "f9de3422b6186747c4d62d906f84c862"
server_key_iv: "be7d52ea1c1527e80cf7da578c9469e1"
server_key_version: "kuduclusterkey@0"
I20260927 15:15:18.005828 22064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "cdc0e9fba29c4ec1b58fab89b5b0155a"
format_stamp: "Formatted at 2026-09-27 15:15:18 on dist-test-slave-f5h6"
server_key: "f9de3422b6186747c4d62d906f84c862"
server_key_iv: "be7d52ea1c1527e80cf7da578c9469e1"
server_key_version: "kuduclusterkey@0"
I20260927 15:15:18.011096 22064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 15:15:18.027573 22079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:15:18.029438 22064 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:15:18.029644 22064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "cdc0e9fba29c4ec1b58fab89b5b0155a"
format_stamp: "Formatted at 2026-09-27 15:15:18 on dist-test-slave-f5h6"
server_key: "f9de3422b6186747c4d62d906f84c862"
server_key_iv: "be7d52ea1c1527e80cf7da578c9469e1"
server_key_version: "kuduclusterkey@0"
I20260927 15:15:18.029932 22064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:15:18.046463 22064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:15:18.050462 22064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:15:18.050706 22064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:15:18.051569 22064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:15:18.052829 22064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:15:18.052899 22064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:15:18.052989 22064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:15:18.053030 22064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:15:18.084165 22064 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:34525
I20260927 15:15:18.084214 22191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:34525 every 8 connection(s)
I20260927 15:15:18.085407 22064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:15:18.091313 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 22064
I20260927 15:15:18.091493 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:15:18.092033 10373 external_mini_cluster.cc:1428] Setting key d3f41e089c324d6deefc07ba45aee248
Sep 27 15:15:18 dist-test-slave-f5h6 krb5kdc[21226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522117, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:15:18.104809 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:15:18.087968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:44071 (local address 127.10.33.126:34031)
0927 15:15:18.088401 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 15:15:18.088406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:15:18.089244 (+ 838us) server_negotiation.cc:407] Connection header received
0927 15:15:18.091072 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:15:18.091077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:15:18.091159 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:15:18.091255 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:15:18.093049 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.093980 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.095662 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.096085 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.096179 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:15:18.099318 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:15:18.099323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:15:18.099326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:15:18.099358 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:15:18.101919 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:15:18.102618 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:15:18.102624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:15:18.102628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:15:18.102751 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:15:18.103139 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:15:18.103145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:15:18.103148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:15:18.103435 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:15:18.103566 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:15:18.104233 (+ 667us) server_negotiation.cc:299] Negotiation successful
0927 15:15:18.104458 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":215,"threads_started":1}
I20260927 15:15:18.105934 22195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:15:18.088435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:34031 (local address 127.10.33.67:44071)
0927 15:15:18.089094 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 15:15:18.089129 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 15:15:18.090763 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:15:18.091544 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:15:18.091556 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:15:18.092115 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:15:18.092863 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:15:18.092887 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.094120 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.094126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:15:18.095529 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:15:18.095541 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.096283 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.096288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:15:18.096490 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:15:18.097119 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:15:18.097141 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:15:18.099100 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:15:18.102053 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:15:18.102063 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:15:18.102081 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:15:18.102503 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:15:18.102850 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:15:18.102855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:15:18.102859 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:15:18.103037 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:15:18.103566 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:15:18.103574 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:15:18.103904 (+ 330us) client_negotiation.cc:769] Sending connection context
0927 15:15:18.104140 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 15:15:18.104645 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":282,"threads_started":1}
I20260927 15:15:18.107326 22192 heartbeater.cc:347] Connected to a master server at 127.10.33.126:34031
I20260927 15:15:18.107630 22192 heartbeater.cc:464] Registering TS with master...
I20260927 15:15:18.108325 22192 heartbeater.cc:511] Master 127.10.33.126:34031 requested a full tablet report, sending...
I20260927 15:15:18.109656 21702 ts_manager.cc:194] Registered new tserver with Master: cdc0e9fba29c4ec1b58fab89b5b0155a (127.10.33.67:34525)
I20260927 15:15:18.110502 21702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:44071
I20260927 15:15:18.119942 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:15:18.137835 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:15:18.124611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48272 (local address 127.10.33.126:34031)
0927 15:15:18.124814 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 15:15:18.124820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:15:18.125402 (+ 582us) server_negotiation.cc:407] Connection header received
0927 15:15:18.125527 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:15:18.125531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:15:18.125630 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:15:18.125729 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:15:18.127141 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.127886 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.129230 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.129731 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.129841 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:15:18.132381 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:15:18.132386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:15:18.132389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:15:18.132423 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:15:18.135087 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:15:18.135883 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:15:18.135888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:15:18.135890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:15:18.136022 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:15:18.136467 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:15:18.136473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:15:18.136476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:15:18.136776 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:15:18.136958 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 15:15:18.137326 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 15:15:18.137489 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:15:18.142210 21702 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:15:18.149490 21702 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:15:18.186949 22207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:15:18.162517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57350 (local address 127.10.33.66:35481)
0927 15:15:18.177035 (+ 14518us) server_negotiation.cc:206] Beginning negotiation
0927 15:15:18.177044 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:15:18.177076 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:15:18.177193 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:15:18.177201 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:15:18.177477 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:15:18.177678 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:15:18.178900 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.180708 (+ 1808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.184504 (+ 3796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.185791 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.185940 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:15:18.186146 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 15:15:18.186472 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 15:15:18.186629 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14356,"thread_start_us":15369,"threads_started":1}
I20260927 15:15:18.186949 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:15:18.162381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:35481 (local address 127.0.0.1:57350)
0927 15:15:18.176108 (+ 13727us) negotiation.cc:107] Waiting for socket to connect
0927 15:15:18.176176 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 15:15:18.176453 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:15:18.178163 (+ 1710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:15:18.178172 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:15:18.178208 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:15:18.178662 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:15:18.178675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.180884 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.180891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:15:18.183520 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:15:18.183537 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.186029 (+ 2492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.186036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:15:18.186226 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:15:18.186267 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 15:15:18.186385 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 15:15:18.186584 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13369,"thread_start_us":14595,"threads_started":1}
I20260927 15:15:18.191855 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:15:18.177737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60838 (local address 127.10.33.67:34525)
0927 15:15:18.184517 (+ 6780us) server_negotiation.cc:206] Beginning negotiation
0927 15:15:18.184525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:15:18.184554 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:15:18.184645 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:15:18.184652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:15:18.184864 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:15:18.185076 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:15:18.186070 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.187856 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.189822 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.190836 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.190951 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:15:18.191170 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 15:15:18.191364 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 15:15:18.191510 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6634,"thread_start_us":6489,"threads_started":1}
I20260927 15:15:18.191884 22209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:15:18.177437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:34525 (local address 127.0.0.1:60838)
0927 15:15:18.183962 (+ 6525us) negotiation.cc:107] Waiting for socket to connect
0927 15:15:18.183979 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:15:18.184092 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:15:18.185431 (+ 1339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:15:18.185435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:15:18.185458 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:15:18.185828 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:15:18.185839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.188010 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.188015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:15:18.189520 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:15:18.189533 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.191025 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.191031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:15:18.191195 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:15:18.191231 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:15:18.191313 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 15:15:18.191445 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6404,"mutex_wait_us":1,"thread_start_us":6718,"threads_started":1}
I20260927 15:15:18.191335 21993 tablet_service.cc:1467] Processing CreateTablet for tablet 63c74ddecde142bfb5f35e8b8b36aa13 (DEFAULT_TABLE table=test-table [id=766a700963ea4434933895b24d54446f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:15:18.192106 22205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:15:18.162437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33257 (local address 127.0.0.1:49508)
0927 15:15:18.176799 (+ 14362us) negotiation.cc:107] Waiting for socket to connect
0927 15:15:18.176815 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:15:18.176924 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:15:18.184759 (+ 7835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:15:18.184764 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:15:18.184787 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:15:18.185150 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:15:18.185161 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.187370 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.187375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:15:18.189387 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:15:18.189403 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.191254 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.191261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:15:18.191431 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:15:18.191463 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 15:15:18.191530 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 15:15:18.191697 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14228,"thread_start_us":13156,"threads_started":1}
I20260927 15:15:18.193178 22208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:15:18.162689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49508 (local address 127.10.33.65:33257)
0927 15:15:18.184118 (+ 21429us) server_negotiation.cc:206] Beginning negotiation
0927 15:15:18.184126 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:15:18.184155 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:15:18.184271 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:15:18.184278 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:15:18.184545 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:15:18.185347 (+ 802us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:15:18.185601 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.187205 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.189612 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.190997 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.192428 (+ 1431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:15:18.192646 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 15:15:18.192719 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 15:15:18.192872 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21247,"thread_start_us":21077,"threads_started":1}
I20260927 15:15:18.193668 21993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c74ddecde142bfb5f35e8b8b36aa13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:15:18.195863 22127 tablet_service.cc:1467] Processing CreateTablet for tablet 63c74ddecde142bfb5f35e8b8b36aa13 (DEFAULT_TABLE table=test-table [id=766a700963ea4434933895b24d54446f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:15:18.196453 21850 tablet_service.cc:1467] Processing CreateTablet for tablet 63c74ddecde142bfb5f35e8b8b36aa13 (DEFAULT_TABLE table=test-table [id=766a700963ea4434933895b24d54446f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:15:18.197304 22127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c74ddecde142bfb5f35e8b8b36aa13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:15:18.197804 21850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c74ddecde142bfb5f35e8b8b36aa13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:15:18.205471 22211 tablet_bootstrap.cc:492] T 63c74ddecde142bfb5f35e8b8b36aa13 P f23df7cb3cc94e0ca160fc2d3c0818d8: Bootstrap starting.
I20260927 15:15:18.208738 22211 tablet_bootstrap.cc:654] T 63c74ddecde142bfb5f35e8b8b36aa13 P f23df7cb3cc94e0ca160fc2d3c0818d8: Neither blocks nor log segments found. Creating new log.
I20260927 15:15:18.209849 22211 log.cc:824] T 63c74ddecde142bfb5f35e8b8b36aa13 P f23df7cb3cc94e0ca160fc2d3c0818d8: Log is configured to *not* fsync() on all Append() calls
I20260927 15:15:18.211869 22214 tablet_bootstrap.cc:492] T 63c74ddecde142bfb5f35e8b8b36aa13 P cdc0e9fba29c4ec1b58fab89b5b0155a: Bootstrap starting.
I20260927 15:15:18.212643 22213 tablet_bootstrap.cc:492] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388: Bootstrap starting.
I20260927 15:15:18.213163 22211 tablet_bootstrap.cc:492] T 63c74ddecde142bfb5f35e8b8b36aa13 P f23df7cb3cc94e0ca160fc2d3c0818d8: No bootstrap required, opened a new log
I20260927 15:15:18.213454 22211 ts_tablet_manager.cc:1397] T 63c74ddecde142bfb5f35e8b8b36aa13 P f23df7cb3cc94e0ca160fc2d3c0818d8: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260927 15:15:18.215684 22214 tablet_bootstrap.cc:654] T 63c74ddecde142bfb5f35e8b8b36aa13 P cdc0e9fba29c4ec1b58fab89b5b0155a: Neither blocks nor log segments found. Creating new log.
I20260927 15:15:18.216580 22213 tablet_bootstrap.cc:654] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388: Neither blocks nor log segments found. Creating new log.
I20260927 15:15:18.216881 22214 log.cc:824] T 63c74ddecde142bfb5f35e8b8b36aa13 P cdc0e9fba29c4ec1b58fab89b5b0155a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:15:18.217779 22213 log.cc:824] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388: Log is configured to *not* fsync() on all Append() calls
I20260927 15:15:18.219872 22214 tablet_bootstrap.cc:492] T 63c74ddecde142bfb5f35e8b8b36aa13 P cdc0e9fba29c4ec1b58fab89b5b0155a: No bootstrap required, opened a new log
I20260927 15:15:18.220132 22214 ts_tablet_manager.cc:1397] T 63c74ddecde142bfb5f35e8b8b36aa13 P cdc0e9fba29c4ec1b58fab89b5b0155a: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260927 15:15:18.220829 22213 tablet_bootstrap.cc:492] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388: No bootstrap required, opened a new log
I20260927 15:15:18.221118 22213 ts_tablet_manager.cc:1397] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260927 15:15:18.223871 22211 raft_consensus.cc:348] T 63c74ddecde142bfb5f35e8b8b36aa13 P f23df7cb3cc94e0ca160fc2d3c0818d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23df7cb3cc94e0ca160fc2d3c0818d8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35481 } } peers { permanent_uuid: "b31c3b6911854dfcbbfbb714e728f388" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33257 } } peers { permanent_uuid: "cdc0e9fba29c4ec1b58fab89b5b0155a" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34525 } }
I20260927 15:15:18.224368 22211 raft_consensus.cc:374] T 63c74ddecde142bfb5f35e8b8b36aa13 P f23df7cb3cc94e0ca160fc2d3c0818d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:15:18.224470 22211 raft_consensus.cc:729] T 63c74ddecde142bfb5f35e8b8b36aa13 P f23df7cb3cc94e0ca160fc2d3c0818d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23df7cb3cc94e0ca160fc2d3c0818d8, State: Initialized, Role: FOLLOWER
I20260927 15:15:18.225262 22211 consensus_queue.cc:260] T 63c74ddecde142bfb5f35e8b8b36aa13 P f23df7cb3cc94e0ca160fc2d3c0818d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23df7cb3cc94e0ca160fc2d3c0818d8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35481 } } peers { permanent_uuid: "b31c3b6911854dfcbbfbb714e728f388" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33257 } } peers { permanent_uuid: "cdc0e9fba29c4ec1b58fab89b5b0155a" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34525 } }
I20260927 15:15:18.228003 22214 raft_consensus.cc:348] T 63c74ddecde142bfb5f35e8b8b36aa13 P cdc0e9fba29c4ec1b58fab89b5b0155a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23df7cb3cc94e0ca160fc2d3c0818d8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35481 } } peers { permanent_uuid: "b31c3b6911854dfcbbfbb714e728f388" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33257 } } peers { permanent_uuid: "cdc0e9fba29c4ec1b58fab89b5b0155a" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34525 } }
I20260927 15:15:18.228421 22214 raft_consensus.cc:374] T 63c74ddecde142bfb5f35e8b8b36aa13 P cdc0e9fba29c4ec1b58fab89b5b0155a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:15:18.228570 22214 raft_consensus.cc:729] T 63c74ddecde142bfb5f35e8b8b36aa13 P cdc0e9fba29c4ec1b58fab89b5b0155a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc0e9fba29c4ec1b58fab89b5b0155a, State: Initialized, Role: FOLLOWER
I20260927 15:15:18.229038 22058 heartbeater.cc:503] Master 127.10.33.126:34031 was elected leader, sending a full tablet report...
I20260927 15:15:18.229254 22214 consensus_queue.cc:260] T 63c74ddecde142bfb5f35e8b8b36aa13 P cdc0e9fba29c4ec1b58fab89b5b0155a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23df7cb3cc94e0ca160fc2d3c0818d8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35481 } } peers { permanent_uuid: "b31c3b6911854dfcbbfbb714e728f388" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33257 } } peers { permanent_uuid: "cdc0e9fba29c4ec1b58fab89b5b0155a" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34525 } }
I20260927 15:15:18.229728 22211 ts_tablet_manager.cc:1428] T 63c74ddecde142bfb5f35e8b8b36aa13 P f23df7cb3cc94e0ca160fc2d3c0818d8: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260927 15:15:18.230304 22192 heartbeater.cc:503] Master 127.10.33.126:34031 was elected leader, sending a full tablet report...
I20260927 15:15:18.230886 22214 ts_tablet_manager.cc:1428] T 63c74ddecde142bfb5f35e8b8b36aa13 P cdc0e9fba29c4ec1b58fab89b5b0155a: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260927 15:15:18.235028 22213 raft_consensus.cc:348] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23df7cb3cc94e0ca160fc2d3c0818d8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35481 } } peers { permanent_uuid: "b31c3b6911854dfcbbfbb714e728f388" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33257 } } peers { permanent_uuid: "cdc0e9fba29c4ec1b58fab89b5b0155a" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34525 } }
I20260927 15:15:18.235646 22213 raft_consensus.cc:374] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:15:18.235767 22213 raft_consensus.cc:729] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b31c3b6911854dfcbbfbb714e728f388, State: Initialized, Role: FOLLOWER
I20260927 15:15:18.236495 22213 consensus_queue.cc:260] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23df7cb3cc94e0ca160fc2d3c0818d8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35481 } } peers { permanent_uuid: "b31c3b6911854dfcbbfbb714e728f388" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33257 } } peers { permanent_uuid: "cdc0e9fba29c4ec1b58fab89b5b0155a" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34525 } }
I20260927 15:15:18.238212 22213 ts_tablet_manager.cc:1428] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388: Time spent starting tablet: real 0.017s user 0.014s sys 0.001s
W20260927 15:15:18.270784 21916 tablet.cc:2369] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:15:18.315239 22059 tablet.cc:2369] T 63c74ddecde142bfb5f35e8b8b36aa13 P f23df7cb3cc94e0ca160fc2d3c0818d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:15:18.337011 22193 tablet.cc:2369] T 63c74ddecde142bfb5f35e8b8b36aa13 P cdc0e9fba29c4ec1b58fab89b5b0155a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:15:18.418368 22219 raft_consensus.cc:482] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:15:18.418707 22219 raft_consensus.cc:504] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23df7cb3cc94e0ca160fc2d3c0818d8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35481 } } peers { permanent_uuid: "b31c3b6911854dfcbbfbb714e728f388" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33257 } } peers { permanent_uuid: "cdc0e9fba29c4ec1b58fab89b5b0155a" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34525 } }
I20260927 15:15:18.420419 22219 leader_election.cc:288] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23df7cb3cc94e0ca160fc2d3c0818d8 (127.10.33.66:35481), cdc0e9fba29c4ec1b58fab89b5b0155a (127.10.33.67:34525)
I20260927 15:15:18.427505 22207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:15:18.420972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:34303 (local address 127.10.33.66:35481)
0927 15:15:18.421173 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 15:15:18.421178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:15:18.421478 (+ 300us) server_negotiation.cc:407] Connection header received
0927 15:15:18.421704 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:15:18.421710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:15:18.421791 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:15:18.421925 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:15:18.422590 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.423623 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.425473 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.426592 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.426709 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:15:18.426763 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 15:15:18.427196 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 15:15:18.427310 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:15:18.427641 22220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:15:18.420708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:35481 (local address 127.10.33.65:34303)
0927 15:15:18.421364 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0927 15:15:18.421383 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:15:18.421537 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:15:18.422062 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:15:18.422067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:15:18.422093 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:15:18.422467 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:15:18.422478 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.423763 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.423768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:15:18.425323 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:15:18.425336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.426811 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.426818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:15:18.427001 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:15:18.427037 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:15:18.427154 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 15:15:18.427292 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":324,"threads_started":1}
I20260927 15:15:18.428507 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:15:18.421729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:40363 (local address 127.10.33.67:34525)
0927 15:15:18.421949 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 15:15:18.421955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:15:18.422220 (+ 265us) server_negotiation.cc:407] Connection header received
0927 15:15:18.422398 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:15:18.422402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:15:18.422457 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:15:18.422561 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:15:18.423551 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.424320 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.426397 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.427535 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.427698 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:15:18.427778 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:15:18.428227 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 15:15:18.428347 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 15:15:18.428550 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:15:18.421484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:34525 (local address 127.10.33.65:40363)
0927 15:15:18.422122 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0927 15:15:18.422149 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:15:18.422277 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:15:18.422989 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:15:18.422994 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:15:18.423021 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:15:18.423384 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:15:18.423395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.424508 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.424512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:15:18.426217 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:15:18.426233 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.427905 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.427911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:15:18.428059 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:15:18.428082 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:15:18.428162 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 15:15:18.428249 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":316,"threads_started":1}
I20260927 15:15:18.428846 22013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c74ddecde142bfb5f35e8b8b36aa13" candidate_uuid: "b31c3b6911854dfcbbfbb714e728f388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23df7cb3cc94e0ca160fc2d3c0818d8" is_pre_election: true
I20260927 15:15:18.429428 22013 raft_consensus.cc:2454] T 63c74ddecde142bfb5f35e8b8b36aa13 P f23df7cb3cc94e0ca160fc2d3c0818d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b31c3b6911854dfcbbfbb714e728f388 in term 0.
I20260927 15:15:18.429507 22147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c74ddecde142bfb5f35e8b8b36aa13" candidate_uuid: "b31c3b6911854dfcbbfbb714e728f388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdc0e9fba29c4ec1b58fab89b5b0155a" is_pre_election: true
I20260927 15:15:18.429997 22147 raft_consensus.cc:2454] T 63c74ddecde142bfb5f35e8b8b36aa13 P cdc0e9fba29c4ec1b58fab89b5b0155a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b31c3b6911854dfcbbfbb714e728f388 in term 0.
I20260927 15:15:18.430188 21805 leader_election.cc:302] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b31c3b6911854dfcbbfbb714e728f388, f23df7cb3cc94e0ca160fc2d3c0818d8; no voters:
I20260927 15:15:18.430610 22219 raft_consensus.cc:2781] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:15:18.430823 22219 raft_consensus.cc:482] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:15:18.430936 22219 raft_consensus.cc:3037] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:15:18.433171 22219 raft_consensus.cc:504] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23df7cb3cc94e0ca160fc2d3c0818d8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35481 } } peers { permanent_uuid: "b31c3b6911854dfcbbfbb714e728f388" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33257 } } peers { permanent_uuid: "cdc0e9fba29c4ec1b58fab89b5b0155a" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34525 } }
I20260927 15:15:18.433758 22219 leader_election.cc:288] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388 [CANDIDATE]: Term 1 election: Requested vote from peers f23df7cb3cc94e0ca160fc2d3c0818d8 (127.10.33.66:35481), cdc0e9fba29c4ec1b58fab89b5b0155a (127.10.33.67:34525)
I20260927 15:15:18.434253 22013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c74ddecde142bfb5f35e8b8b36aa13" candidate_uuid: "b31c3b6911854dfcbbfbb714e728f388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23df7cb3cc94e0ca160fc2d3c0818d8"
I20260927 15:15:18.434283 22147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c74ddecde142bfb5f35e8b8b36aa13" candidate_uuid: "b31c3b6911854dfcbbfbb714e728f388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdc0e9fba29c4ec1b58fab89b5b0155a"
I20260927 15:15:18.434468 22013 raft_consensus.cc:3037] T 63c74ddecde142bfb5f35e8b8b36aa13 P f23df7cb3cc94e0ca160fc2d3c0818d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:15:18.434494 22147 raft_consensus.cc:3037] T 63c74ddecde142bfb5f35e8b8b36aa13 P cdc0e9fba29c4ec1b58fab89b5b0155a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:15:18.437578 22013 raft_consensus.cc:2454] T 63c74ddecde142bfb5f35e8b8b36aa13 P f23df7cb3cc94e0ca160fc2d3c0818d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b31c3b6911854dfcbbfbb714e728f388 in term 1.
I20260927 15:15:18.437580 22147 raft_consensus.cc:2454] T 63c74ddecde142bfb5f35e8b8b36aa13 P cdc0e9fba29c4ec1b58fab89b5b0155a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b31c3b6911854dfcbbfbb714e728f388 in term 1.
I20260927 15:15:18.437988 21805 leader_election.cc:302] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b31c3b6911854dfcbbfbb714e728f388, f23df7cb3cc94e0ca160fc2d3c0818d8; no voters:
I20260927 15:15:18.438256 22219 raft_consensus.cc:2781] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:15:18.438658 22219 raft_consensus.cc:686] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388 [term 1 LEADER]: Becoming Leader. State: Replica: b31c3b6911854dfcbbfbb714e728f388, State: Running, Role: LEADER
I20260927 15:15:18.439087 22219 consensus_queue.cc:237] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23df7cb3cc94e0ca160fc2d3c0818d8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35481 } } peers { permanent_uuid: "b31c3b6911854dfcbbfbb714e728f388" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33257 } } peers { permanent_uuid: "cdc0e9fba29c4ec1b58fab89b5b0155a" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34525 } }
I20260927 15:15:18.443395 21702 catalog_manager.cc:5495] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388 reported cstate change: term changed from 0 to 1, leader changed from <none> to b31c3b6911854dfcbbfbb714e728f388 (127.10.33.65). New cstate: current_term: 1 leader_uuid: "b31c3b6911854dfcbbfbb714e728f388" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23df7cb3cc94e0ca160fc2d3c0818d8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b31c3b6911854dfcbbfbb714e728f388" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdc0e9fba29c4ec1b58fab89b5b0155a" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20260927 15:15:18.494079 22208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:15:18.488745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49518 (local address 127.10.33.65:33257)
0927 15:15:18.488973 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 15:15:18.488978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:15:18.488996 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:15:18.489162 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:15:18.489167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:15:18.489254 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:15:18.489389 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:15:18.489884 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.490666 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.491996 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.492380 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.492471 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:15:18.493014 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:15:18.493553 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:15:18.493659 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 15:15:18.493796 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 15:15:18.493886 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260927 15:15:18.502254 22147 raft_consensus.cc:1261] T 63c74ddecde142bfb5f35e8b8b36aa13 P cdc0e9fba29c4ec1b58fab89b5b0155a [term 1 FOLLOWER]: Refusing update from remote peer b31c3b6911854dfcbbfbb714e728f388: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:15:18.502233 22013 raft_consensus.cc:1261] T 63c74ddecde142bfb5f35e8b8b36aa13 P f23df7cb3cc94e0ca160fc2d3c0818d8 [term 1 FOLLOWER]: Refusing update from remote peer b31c3b6911854dfcbbfbb714e728f388: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:15:18.503212 22222 consensus_queue.cc:1035] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc0e9fba29c4ec1b58fab89b5b0155a" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:15:18.503939 22219 consensus_queue.cc:1035] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23df7cb3cc94e0ca160fc2d3c0818d8" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:15:18.531033 21701 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:15:18.531314 21701 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:15:18.535069 21701 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94295f3a55f348c1be8f2a0c224d618e: Sending DeleteTablet for 3 replicas of tablet 63c74ddecde142bfb5f35e8b8b36aa13
I20260927 15:15:18.536271 21850 tablet_service.cc:1514] Processing DeleteTablet for tablet 63c74ddecde142bfb5f35e8b8b36aa13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:15:18 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:49508
I20260927 15:15:18.537016 21993 tablet_service.cc:1514] Processing DeleteTablet for tablet 63c74ddecde142bfb5f35e8b8b36aa13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:15:18 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:57350
I20260927 15:15:18.537168 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 21782
I20260927 15:15:18.538062 22127 tablet_service.cc:1514] Processing DeleteTablet for tablet 63c74ddecde142bfb5f35e8b8b36aa13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:15:18 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:60838
I20260927 15:15:18.548884 22235 tablet_replica.cc:330] T 63c74ddecde142bfb5f35e8b8b36aa13 P f23df7cb3cc94e0ca160fc2d3c0818d8: stopping tablet replica
I20260927 15:15:18.549443 22235 raft_consensus.cc:2229] T 63c74ddecde142bfb5f35e8b8b36aa13 P f23df7cb3cc94e0ca160fc2d3c0818d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:15:18.550041 22235 raft_consensus.cc:2258] T 63c74ddecde142bfb5f35e8b8b36aa13 P f23df7cb3cc94e0ca160fc2d3c0818d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:15:18.550971 22237 tablet_replica.cc:330] T 63c74ddecde142bfb5f35e8b8b36aa13 P cdc0e9fba29c4ec1b58fab89b5b0155a: stopping tablet replica
I20260927 15:15:18.551517 22237 raft_consensus.cc:2229] T 63c74ddecde142bfb5f35e8b8b36aa13 P cdc0e9fba29c4ec1b58fab89b5b0155a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:15:18.552053 22237 raft_consensus.cc:2258] T 63c74ddecde142bfb5f35e8b8b36aa13 P cdc0e9fba29c4ec1b58fab89b5b0155a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:15:18.553581 22235 ts_tablet_manager.cc:1905] T 63c74ddecde142bfb5f35e8b8b36aa13 P f23df7cb3cc94e0ca160fc2d3c0818d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:15:18.554193 22237 ts_tablet_manager.cc:1905] T 63c74ddecde142bfb5f35e8b8b36aa13 P cdc0e9fba29c4ec1b58fab89b5b0155a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:15:18.556751 22234 tablet_replica.cc:330] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388: stopping tablet replica
I20260927 15:15:18.557250 22234 raft_consensus.cc:2229] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:15:18.558122 22234 raft_consensus.cc:2258] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:15:18.560539 22235 ts_tablet_manager.cc:1918] T 63c74ddecde142bfb5f35e8b8b36aa13 P f23df7cb3cc94e0ca160fc2d3c0818d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:15:18.560706 22235 log.cc:1196] T 63c74ddecde142bfb5f35e8b8b36aa13 P f23df7cb3cc94e0ca160fc2d3c0818d8: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/63c74ddecde142bfb5f35e8b8b36aa13
I20260927 15:15:18.561178 22235 ts_tablet_manager.cc:1939] T 63c74ddecde142bfb5f35e8b8b36aa13 P f23df7cb3cc94e0ca160fc2d3c0818d8: Deleting consensus metadata
I20260927 15:15:18.563192 21688 catalog_manager.cc:4841] TS f23df7cb3cc94e0ca160fc2d3c0818d8 (127.10.33.66:35481): tablet 63c74ddecde142bfb5f35e8b8b36aa13 (table test-table [id=766a700963ea4434933895b24d54446f]) successfully deleted
I20260927 15:15:18.565068 22234 ts_tablet_manager.cc:1905] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 15:15:18 dist-test-slave-f5h6 krb5kdc[21226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.65@KRBTEST.COM
I20260927 15:15:18.573011 22234 ts_tablet_manager.cc:1918] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:15:18.573212 22234 log.cc:1196] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/63c74ddecde142bfb5f35e8b8b36aa13
I20260927 15:15:18.573787 22234 ts_tablet_manager.cc:1939] T 63c74ddecde142bfb5f35e8b8b36aa13 P b31c3b6911854dfcbbfbb714e728f388: Deleting consensus metadata
I20260927 15:15:18.574481 22237 ts_tablet_manager.cc:1918] T 63c74ddecde142bfb5f35e8b8b36aa13 P cdc0e9fba29c4ec1b58fab89b5b0155a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:15:18.574664 22237 log.cc:1196] T 63c74ddecde142bfb5f35e8b8b36aa13 P cdc0e9fba29c4ec1b58fab89b5b0155a: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/63c74ddecde142bfb5f35e8b8b36aa13
I20260927 15:15:18.575153 22237 ts_tablet_manager.cc:1939] T 63c74ddecde142bfb5f35e8b8b36aa13 P cdc0e9fba29c4ec1b58fab89b5b0155a: Deleting consensus metadata
I20260927 15:15:18.575714 21687 catalog_manager.cc:4841] TS b31c3b6911854dfcbbfbb714e728f388 (127.10.33.65:33257): tablet 63c74ddecde142bfb5f35e8b8b36aa13 (table test-table [id=766a700963ea4434933895b24d54446f]) successfully deleted
I20260927 15:15:18.576890 21688 catalog_manager.cc:4841] TS cdc0e9fba29c4ec1b58fab89b5b0155a (127.10.33.67:34525): tablet 63c74ddecde142bfb5f35e8b8b36aa13 (table test-table [id=766a700963ea4434933895b24d54446f]) successfully deleted
I20260927 15:15:18.578408 22208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:15:18.538431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49534 (local address 127.10.33.65:33257)
0927 15:15:18.539752 (+ 1321us) server_negotiation.cc:206] Beginning negotiation
0927 15:15:18.539758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:15:18.551646 (+ 11888us) server_negotiation.cc:407] Connection header received
0927 15:15:18.552575 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:15:18.552580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:15:18.552690 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:15:18.552777 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:15:18.558599 (+ 5822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.559755 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.561909 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.562542 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.562654 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:15:18.570968 (+ 8314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:15:18.570988 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:15:18.571012 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:15:18.571060 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:15:18.575000 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:15:18.575950 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:15:18.575959 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:15:18.575962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:15:18.576137 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:15:18.576529 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:15:18.576534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:15:18.576537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:15:18.576985 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:15:18.577226 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 15:15:18.577711 (+ 485us) server_negotiation.cc:299] Negotiation successful
0927 15:15:18.577965 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1168}
I20260927 15:15:18.579053 21910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:15:18.678071 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 21782
I20260927 15:15:18.693523 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 21921
Sep 27 15:15:18 dist-test-slave-f5h6 krb5kdc[21226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.66@KRBTEST.COM
I20260927 15:15:18.708431 22207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:15:18.694289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.10.33.66:35481)
0927 15:15:18.694462 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 15:15:18.694468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:15:18.694525 (+ 57us) server_negotiation.cc:407] Connection header received
0927 15:15:18.694723 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:15:18.694728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:15:18.694820 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:15:18.694903 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:15:18.696135 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.696849 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.698138 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.698527 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.698639 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:15:18.702939 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:15:18.702954 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:15:18.702971 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:15:18.703005 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:15:18.705803 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:15:18.706588 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:15:18.706596 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:15:18.706598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:15:18.706736 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:15:18.707109 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:15:18.707112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:15:18.707113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:15:18.707479 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:15:18.707693 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 15:15:18.707964 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 15:15:18.708138 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:15:18.709092 22053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:15:18.798275 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 21921
I20260927 15:15:18.811092 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 22064
Sep 27 15:15:18 dist-test-slave-f5h6 krb5kdc[21226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.67@KRBTEST.COM
I20260927 15:15:18.827142 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:15:18.811897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60850 (local address 127.10.33.67:34525)
0927 15:15:18.812064 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 15:15:18.812069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:15:18.812088 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:15:18.812251 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:15:18.812255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:15:18.812328 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:15:18.812414 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:15:18.813716 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.814469 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.815821 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:15:18.816202 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:15:18.816310 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:15:18.821189 (+ 4879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:15:18.821205 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:15:18.821222 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:15:18.821284 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:15:18.824032 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:15:18.824742 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:15:18.824750 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:15:18.824753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:15:18.824886 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:15:18.825324 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:15:18.825329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:15:18.825332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:15:18.825895 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:15:18.826213 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 15:15:18.826499 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 15:15:18.826755 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:15:18.827853 22187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:15:18.922534 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 22064
I20260927 15:15:18.935497 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 21670
I20260927 15:15:18.936785 21732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:15:19.048027 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 21670
I20260927 15:15:19.069332 10373 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 15:15:19.405732 10373 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 15:15:19.406014 10373 mini_ranger.cc:66] Stopping Ranger...
I20260927 15:15:19.742372 10373 mini_ranger.cc:68] Stopped Ranger
2026-09-27 15:15:19.742 UTC [21256] LOG: received smart shutdown request
2026-09-27 15:15:19.746 UTC [21256] LOG: background worker "logical replication launcher" (PID 21265) exited with exit code 1
2026-09-27 15:15:19.746 UTC [21260] LOG: shutting down
2026-09-27 15:15:19.940 UTC [21256] LOG: database system is shut down
2026-09-27T15:15:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50317 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:15:20 dist-test-slave-f5h6 krb5kdc[22250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:15:20 dist-test-slave-f5h6 krb5kdc[22250](info): set up 2 sockets
Sep 27 15:15:20 dist-test-slave-f5h6 krb5kdc[22250](info): commencing operation
krb5kdc: starting...
W20260927 15:15:21.078169 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:15:21 dist-test-slave-f5h6 krb5kdc[22250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:15:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:15:21Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.33.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.33.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.33.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/rangeradmin_127.10.33.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.33.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/rangeradmin_127.10.33.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.33.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.33.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.33.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/rangerlookup_127.10.33.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.33.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/rangerlookup_127.10.33.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.33.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.33.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.33.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/HTTP_127.10.33.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.33.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/HTTP_127.10.33.84@KRBTEST.COM.keytab.
I20260927 15:15:21.389075 10373 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskQegxT3/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/postgres -l logfile start
2026-09-27 15:15:24.112 UTC [22280] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 15:15:24.112 UTC [22280] LOG: listening on IPv4 address "127.10.33.84", port 41327
2026-09-27 15:15:24.116 UTC [22280] LOG: listening on Unix socket "/tmp/.s.PGSQL.41327"
2026-09-27 15:15:24.136 UTC [22283] LOG: database system was shut down at 2026-09-27 15:15:22 UTC
2026-09-27 15:15:24.139 UTC [22280] LOG: database system is ready to accept connections
I20260927 15:15:25.033700 10373 mini_postgres.cc:96] Postgres bound to 41327
2026-09-27 15:15:25.049 UTC [22291] FATAL: database "slave" does not exist
127.10.33.84:41327 - accepting connections
I20260927 15:15:25.049618 10373 mini_ranger.cc:161] Starting Ranger...
I20260927 15:15:25.074617 10373 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 15:15:25.612610 10373 mini_ranger.cc:87] Created ranger Postgres database
I20260927 15:15:25.612725 10373 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ranger-admin
2026-09-27 15:15:25,897 [I] DB FLAVOR :POSTGRES
2026-09-27 15:15:25,897 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:15:25,897 [I] Checking connection..
2026-09-27 15:15:25,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 15:15:26,141 [I] Checking connection passed.
2026-09-27 15:15:26,142 [I] --------- Verifying version history table ---------
2026-09-27 15:15:26,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:15:26,410 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:15:26,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:15:26,668 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:15:26,669 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 15:15:26,669 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 15:15:26.913 UTC [22355] WARNING: there is no transaction in progress
2026-09-27 15:15:26,919 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 15:15:26,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:15:27,190 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 15:15:27,190 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:15:27,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 15:15:27,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 15:15:27,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f5h6.c.gcp-upstream.internal','N') ;"
2026-09-27 15:15:27,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:15:28,195 [I] Table x_portal_user does not exist in database ranger
2026-09-27 15:15:28,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:15:28,442 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 15:15:28,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:15:28,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:15:28,955 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 15:15:28,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 15:15:29.978 UTC [22463] WARNING: there is no transaction in progress
2026-09-27 15:15:29.997 UTC [22463] WARNING: there is no transaction in progress
2026-09-27 15:15:30.018 UTC [22463] WARNING: there is no transaction in progress
2026-09-27 15:15:30.036 UTC [22463] WARNING: there is no transaction in progress
2026-09-27 15:15:30.056 UTC [22463] WARNING: there is no transaction in progress
2026-09-27 15:15:30.076 UTC [22463] WARNING: there is no transaction in progress
2026-09-27 15:15:30.141 UTC [22463] WARNING: there is no transaction in progress
2026-09-27 15:15:30.150 UTC [22463] WARNING: there is no transaction in progress
2026-09-27 15:15:30.168 UTC [22463] WARNING: there is no transaction in progress
2026-09-27 15:15:30.180 UTC [22463] WARNING: there is no transaction in progress
2026-09-27 15:15:30.193 UTC [22463] WARNING: there is no transaction in progress
2026-09-27 15:15:30.209 UTC [22463] WARNING: there is no transaction in progress
2026-09-27 15:15:30.220 UTC [22463] WARNING: there is no transaction in progress
2026-09-27 15:15:30.227 UTC [22463] WARNING: there is no transaction in progress
2026-09-27 15:15:30.237 UTC [22463] WARNING: there is no transaction in progress
2026-09-27 15:15:30.245 UTC [22463] WARNING: there is no transaction in progress
2026-09-27 15:15:30,975 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 15:15:30,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 15:15:31,230 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 15:15:31,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:15:31,494 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 15:15:31,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:15:31,747 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 15:15:31,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:15:32,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:15:32,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f5h6.c.gcp-upstream.internal';"
2026-09-27 15:15:32,492 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 15:15:32,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 15:15:32,752 [I] DB_PATCHES have already been applied
I20260927 15:15:32.758564 10373 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ranger-admin:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskQegxT3/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 15:15:32.758664 10373 mini_ranger.cc:193] Using host: 127.10.33.84
I20260927 15:15:32.772538 10373 mini_ranger.cc:227] Ranger admin URL: http://127.10.33.84:33975
Sep 27, 2026 3:15:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:15:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:15:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 3:15:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 3:15:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 3:15:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 3:15:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 15:15:33.834757 10373 mini_ranger.cc:160] Time spent starting Ranger: real 8.785s user 0.005s sys 0.033s
Sep 27, 2026 3:15:55 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 3:15:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 3:15:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 3:15:55 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 3:15:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 3:15:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 3:15:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 3:15:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 3:15:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 3:15:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 3:15:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 3:15:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 3:15:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 3:15:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 3:15:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 3:15:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 3:15:55 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 3:15:56 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 15:15:56.862182 10373 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.33.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.33.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.33.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/rangerkms_127.10.33.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.33.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/rangerkms_127.10.33.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.33.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.33.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.33.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/HTTP_127.10.33.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.33.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/HTTP_127.10.33.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 15:15:57 dist-test-slave-f5h6 krb5kdc[22250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522157, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 15:15:57.027019 10373 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 15:15:57.047363 10373 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 15:15:58.135089 10373 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 15:15:58.135192 10373 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ranger-kms
I20260927 15:15:58.135231 10373 mini_ranger_kms.cc:227] Using postgres at 127.10.33.84:41327
2026-09-27 15:15:58,456 [I] DB FLAVOR :POSTGRES
2026-09-27 15:15:58,456 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:15:58,456 [I] Checking connection
2026-09-27 15:15:58,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 15:15:58,692 [I] connection success
2026-09-27 15:15:58,693 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 15:15:58,693 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 15:15:58,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 15:15:58,933 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 15:15:58,933 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:15:58,934 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 15:15:58,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.33.84:41327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 15:15:59,198 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 15:15:59.701707 10373 mini_ranger_kms.cc:310] Created kms service
I20260927 15:15:59.909188 10373 mini_ranger_kms.cc:326] Created kudu user
I20260927 15:15:59.956126 10373 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 15:15:59 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522157, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
I20260927 15:16:00.364097 10373 mini_ranger_kms.cc:384] Added ranger policy
I20260927 15:16:00.364439 10373 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ranger-kms:/tmp/dist-test-taskQegxT3/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskQegxT3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 15:16:00.364507 10373 mini_ranger_kms.cc:240] Using host: 127.10.33.84
I20260927 15:16:00.381309 10373 mini_ranger_kms.cc:276] Ranger KMS PID: 22644
I20260927 15:16:00.381441 10373 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.33.84:57123
Sep 27, 2026 3:16:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:16:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:16:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 3:16:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 3:16:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 3:16:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 15:16:02.481722 10373 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.455s user 0.059s sys 0.280s
I20260927 15:16:02.481997 10373 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 15:16:02.482173 10373 mini_ranger_kms.cc:401] 127.10.33.84:57123/kms/v1/keys
Sep 27 15:16:02 dist-test-slave-f5h6 krb5kdc[22250](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790522162, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.33.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 15:16:03 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522162, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.33.84@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
Sep 27 15:16:03 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522162, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.33.84@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
Sep 27 15:16:05 dist-test-slave-f5h6 krb5kdc[22250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:16:05.890513 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:35377
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:35377
--ranger_config_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.33.84:57123/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:06.218348 22694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:06.218690 22694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:06.218822 22694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:06.226321 22694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:06.226467 22694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:06.226544 22694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:06.226615 22694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:06.226679 22694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:16:06.237008 22694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.33.84:57123/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:35377
--ranger_config_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:35377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:06.238405 22694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:06.239724 22694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:06.251950 22700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:06.252030 22694 server_base.cc:1034] running on GCE node
W20260927 15:16:06.251982 22699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:06.251914 22702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:06.253089 22694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:06.254664 22694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:06.255944 22694 hybrid_clock.cc:648] HybridClock initialized: now 1790522166255870 us; error 51 us; skew 500 ppm
I20260927 15:16:06.256331 22694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:06 dist-test-slave-f5h6 krb5kdc[22250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522166, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:06.265228 22694 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:16:06.266861 22694 webserver.cc:460] Webserver started at http://127.10.33.126:40363/ using document root <none> and password file <none>
I20260927 15:16:06.267546 22694 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:06.267649 22694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:06.267952 22694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:16:06 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522166, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
I20260927 15:16:06.317361 22694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "d2baac03b77044029c44e1dd64283ff5"
format_stamp: "Formatted at 2026-09-27 15:16:06 on dist-test-slave-f5h6"
server_key: "360b641bda968d47e18c6f88a0f11e98"
server_key_iv: "8cbdd550b525666689079d32ffb26df4"
server_key_version: "kuduclusterkey@0"
I20260927 15:16:06.318260 22694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "d2baac03b77044029c44e1dd64283ff5"
format_stamp: "Formatted at 2026-09-27 15:16:06 on dist-test-slave-f5h6"
server_key: "360b641bda968d47e18c6f88a0f11e98"
server_key_iv: "8cbdd550b525666689079d32ffb26df4"
server_key_version: "kuduclusterkey@0"
I20260927 15:16:06.323587 22694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:16:06.340246 22709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:06.341979 22694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:16:06.342160 22694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "d2baac03b77044029c44e1dd64283ff5"
format_stamp: "Formatted at 2026-09-27 15:16:06 on dist-test-slave-f5h6"
server_key: "360b641bda968d47e18c6f88a0f11e98"
server_key_iv: "8cbdd550b525666689079d32ffb26df4"
server_key_version: "kuduclusterkey@0"
I20260927 15:16:06.342403 22694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:06.362398 22694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:06.366545 22694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:06.366830 22694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:06.384160 22694 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:35377
I20260927 15:16:06.384162 22760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:35377 every 8 connection(s)
I20260927 15:16:06.385571 22694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:16:06.389284 22761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:06.394423 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 22694
I20260927 15:16:06.394644 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:16:06.395386 10373 external_mini_cluster.cc:1428] Setting key 1c214e31f0bca76dcba645a28adb34b2
I20260927 15:16:06.400933 22761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5: Bootstrap starting.
I20260927 15:16:06.403944 22761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:06.404891 22761 log.cc:824] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:06.407856 22761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5: No bootstrap required, opened a new log
Sep 27 15:16:06 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:16:06.415764 22761 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2baac03b77044029c44e1dd64283ff5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35377 } }
I20260927 15:16:06.416157 22761 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:06.416292 22761 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2baac03b77044029c44e1dd64283ff5, State: Initialized, Role: FOLLOWER
I20260927 15:16:06.416937 22761 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2baac03b77044029c44e1dd64283ff5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35377 } }
I20260927 15:16:06.417166 22761 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:16:06.417294 22761 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:16:06.417454 22761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:06.417627 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:06.397781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55528 (local address 127.10.33.126:35377)
0927 15:16:06.398708 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:06.398721 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:06.398763 (+ 42us) server_negotiation.cc:407] Connection header received
0927 15:16:06.400459 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:06.400481 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:06.400873 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:06.401253 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:06.402657 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:06.403905 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:06.405179 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:06.406217 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:06.406409 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:06.410144 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:06.410152 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:06.410168 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:06.410205 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:06.413332 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:06.414097 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:06.414104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:06.414111 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:06.414280 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:06.414653 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:06.414657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:06.414659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:06.415236 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:06.415522 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:06.415968 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 15:16:06.416549 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":402,"threads_started":1}
I20260927 15:16:06.419798 22761 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2baac03b77044029c44e1dd64283ff5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35377 } }
I20260927 15:16:06.420277 22761 leader_election.cc:302] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2baac03b77044029c44e1dd64283ff5; no voters:
I20260927 15:16:06.420796 22761 leader_election.cc:288] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:16:06.420913 22766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:06.421507 22766 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [term 1 LEADER]: Becoming Leader. State: Replica: d2baac03b77044029c44e1dd64283ff5, State: Running, Role: LEADER
I20260927 15:16:06.422108 22766 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2baac03b77044029c44e1dd64283ff5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35377 } }
I20260927 15:16:06.422721 22761 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:06.424502 22767 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2baac03b77044029c44e1dd64283ff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2baac03b77044029c44e1dd64283ff5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35377 } } }
I20260927 15:16:06.426975 22767 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:06.424871 22768 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2baac03b77044029c44e1dd64283ff5. Latest consensus state: current_term: 1 leader_uuid: "d2baac03b77044029c44e1dd64283ff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2baac03b77044029c44e1dd64283ff5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35377 } } }
I20260927 15:16:06.428184 22768 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:06.430486 22775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:16:06.437610 22775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:16:06.443886 22761 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 15:16:06.456529 22775 catalog_manager.cc:1347] Generated new cluster ID: 9d6ff55959f1488fbfd8315cab6453b1
I20260927 15:16:06.456780 22775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:16:06.475723 22775 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:16:06.477733 22775 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:16:06.494971 22775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5: Generated new TSK 0
I20260927 15:16:06.496146 22775 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.33.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
I20260927 15:16:07.208259 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.33.84:57123/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:35377
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:16:07 dist-test-slave-f5h6 krb5kdc[22250](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790522167, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 15:16:07.594625 22806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:07.595036 22806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:07.595249 22806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:07.606907 22806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:07.607095 22806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:07.607336 22806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:16:07.620019 22806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.33.84:57123/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:35377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:07.621913 22806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:07.623647 22806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:07.640774 22816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:07.642318 22817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:07.659818 22819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:07.660844 22806 server_base.cc:1034] running on GCE node
I20260927 15:16:07.661782 22806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:07.665618 22806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:07.667384 22806 hybrid_clock.cc:648] HybridClock initialized: now 1790522167667321 us; error 58 us; skew 500 ppm
I20260927 15:16:07.667886 22806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:07 dist-test-slave-f5h6 krb5kdc[22250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522167, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:07.672628 22806 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:16:07.676185 22806 webserver.cc:460] Webserver started at http://127.10.33.65:35073/ using document root <none> and password file <none>
I20260927 15:16:07.677287 22806 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:07.677503 22806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:07.677953 22806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:16:07 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522167, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
I20260927 15:16:07.702098 22806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "dc57b1c5139f42249a8d061b9b3ade06"
format_stamp: "Formatted at 2026-09-27 15:16:07 on dist-test-slave-f5h6"
server_key: "102e4f3936b95d95a8c984533454ba50"
server_key_iv: "31193d4391def16cd36bcd201c67d28f"
server_key_version: "kuduclusterkey@0"
I20260927 15:16:07.703372 22806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "dc57b1c5139f42249a8d061b9b3ade06"
format_stamp: "Formatted at 2026-09-27 15:16:07 on dist-test-slave-f5h6"
server_key: "102e4f3936b95d95a8c984533454ba50"
server_key_iv: "31193d4391def16cd36bcd201c67d28f"
server_key_version: "kuduclusterkey@0"
I20260927 15:16:07.726202 22806 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.010s sys 0.012s
I20260927 15:16:07.748377 22826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:07.750386 22806 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:16:07.750684 22806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "dc57b1c5139f42249a8d061b9b3ade06"
format_stamp: "Formatted at 2026-09-27 15:16:07 on dist-test-slave-f5h6"
server_key: "102e4f3936b95d95a8c984533454ba50"
server_key_iv: "31193d4391def16cd36bcd201c67d28f"
server_key_version: "kuduclusterkey@0"
I20260927 15:16:07.751078 22806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:07.785661 22806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:07.791030 22806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:07.791441 22806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:07.799015 22806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:07.803778 22806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:07.803889 22806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:07.804042 22806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:07.804105 22806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:07.968690 22806 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:39617
I20260927 15:16:07.970621 22806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:16:07.968706 22938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:39617 every 8 connection(s)
I20260927 15:16:07.978832 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 22806
I20260927 15:16:07.979051 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:16:07.979818 10373 external_mini_cluster.cc:1428] Setting key 3a0465131c9377bf82e3ae791e7e907a
WARNING: no policy specified for kudu/127.10.33.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.66@KRBTEST.COM" created.
Sep 27 15:16:08 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522167, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:16:08.071034 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:08.001082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:48633 (local address 127.10.33.126:35377)
0927 15:16:08.026681 (+ 25599us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:08.026689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:08.029934 (+ 3245us) server_negotiation.cc:407] Connection header received
0927 15:16:08.035944 (+ 6010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:08.035952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:08.036075 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:08.036155 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:08.042980 (+ 6825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:08.044164 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:08.049760 (+ 5596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:08.050450 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:08.050572 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:08.060620 (+ 10048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:08.060628 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:08.060633 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:08.060695 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:08.064694 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:08.066928 (+ 2234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:08.066936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:08.066939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:08.067141 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:08.068212 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:08.068218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:08.068222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:08.068856 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:08.069004 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:08.070139 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0927 15:16:08.070425 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25426,"thread_start_us":25612,"threads_started":1}
I20260927 15:16:08.074136 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:08.000337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35377 (local address 127.10.33.65:48633)
0927 15:16:08.027484 (+ 27147us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:08.027549 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:08.035528 (+ 7979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:08.036788 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:08.036802 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:08.037528 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:08.042672 (+ 5144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:08.042713 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:08.046617 (+ 3904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:08.046625 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:08.049523 (+ 2898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:08.049550 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:08.051003 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:08.051010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:08.051307 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:08.052739 (+ 1432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:08.052779 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:08.060260 (+ 7481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:08.065998 (+ 5738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:08.066010 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:08.066033 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:08.066724 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:08.067736 (+ 1012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:08.067742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:08.067746 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:08.068058 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:08.069117 (+ 1059us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:08.069127 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:08.069641 (+ 514us) client_negotiation.cc:769] Sending connection context
0927 15:16:08.071626 (+ 1985us) client_negotiation.cc:240] Negotiation successful
0927 15:16:08.072377 (+ 751us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26756,"thread_start_us":33419,"threads_started":1}
I20260927 15:16:08.076462 22939 heartbeater.cc:347] Connected to a master server at 127.10.33.126:35377
I20260927 15:16:08.076789 22939 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:08.077632 22939 heartbeater.cc:511] Master 127.10.33.126:35377 requested a full tablet report, sending...
I20260927 15:16:08.080351 22726 ts_manager.cc:194] Registered new tserver with Master: dc57b1c5139f42249a8d061b9b3ade06 (127.10.33.65:39617)
I20260927 15:16:08.082595 22726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:48633
Sep 27 15:16:08 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522167, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
W20260927 15:16:08.136610 10373 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.10.33.66: real 0.153s user 0.000s sys 0.017s
I20260927 15:16:08.137521 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.33.84:57123/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:35377
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:16:08 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522167, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
W20260927 15:16:08.617344 22945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:08.617654 22945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:08.617806 22945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:08.626621 22945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:08.626748 22945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:08.626904 22945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:16:08.640926 22945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.33.84:57123/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:35377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:08.642875 22945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:08.644502 22945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:08.661922 22955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:08.661928 22954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:08.665267 22959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:08.665609 22945 server_base.cc:1034] running on GCE node
I20260927 15:16:08.666304 22945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:08.667434 22945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:08.668660 22945 hybrid_clock.cc:648] HybridClock initialized: now 1790522168668597 us; error 59 us; skew 500 ppm
I20260927 15:16:08.668991 22945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:08 dist-test-slave-f5h6 krb5kdc[22250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522168, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:08.672187 22945 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:16:08.673820 22945 webserver.cc:460] Webserver started at http://127.10.33.66:43835/ using document root <none> and password file <none>
I20260927 15:16:08.674599 22945 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:08.674710 22945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:08.674995 22945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:16:08 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522168, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
I20260927 15:16:08.693037 22945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "1169cea5eb7b4fa2a4256c321ab5e016"
format_stamp: "Formatted at 2026-09-27 15:16:08 on dist-test-slave-f5h6"
server_key: "b6a577e3d0867299df1bbfc8caf9efa7"
server_key_iv: "edab07d4b722eb862b2d81642eed0aa2"
server_key_version: "kuduclusterkey@0"
I20260927 15:16:08.694245 22945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "1169cea5eb7b4fa2a4256c321ab5e016"
format_stamp: "Formatted at 2026-09-27 15:16:08 on dist-test-slave-f5h6"
server_key: "b6a577e3d0867299df1bbfc8caf9efa7"
server_key_iv: "edab07d4b722eb862b2d81642eed0aa2"
server_key_version: "kuduclusterkey@0"
I20260927 15:16:08.701048 22945 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 15:16:08.715508 22969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:08.716975 22945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:16:08.717146 22945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "1169cea5eb7b4fa2a4256c321ab5e016"
format_stamp: "Formatted at 2026-09-27 15:16:08 on dist-test-slave-f5h6"
server_key: "b6a577e3d0867299df1bbfc8caf9efa7"
server_key_iv: "edab07d4b722eb862b2d81642eed0aa2"
server_key_version: "kuduclusterkey@0"
I20260927 15:16:08.717374 22945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:08.738667 22945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:08.742812 22945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:08.743090 22945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:08.744107 22945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:08.745520 22945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:08.745597 22945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:08.745713 22945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:08.745762 22945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:08.782579 22945 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:39089
I20260927 15:16:08.782595 23081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:39089 every 8 connection(s)
I20260927 15:16:08.783866 22945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:16:08.791971 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 22945
I20260927 15:16:08.792146 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:16:08.792614 10373 external_mini_cluster.cc:1428] Setting key 9c8f5dc9faac58b3f53195e2e0d3c58d
Sep 27 15:16:08 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522168, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:16:08.806239 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:08.786516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:43737 (local address 127.10.33.126:35377)
0927 15:16:08.787029 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:08.787034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:08.788130 (+ 1096us) server_negotiation.cc:407] Connection header received
0927 15:16:08.790010 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:08.790015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:08.790111 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:08.790204 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:08.791594 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:08.792639 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:08.795077 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:08.795617 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:08.795743 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:08.799841 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:08.799847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:08.799851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:08.799890 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:08.802601 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:08.803535 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:08.803541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:08.803544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:08.803690 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:08.804248 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:08.804254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:08.804257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:08.804599 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:08.804729 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:08.805635 (+ 906us) server_negotiation.cc:299] Negotiation successful
0927 15:16:08.805883 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":268,"threads_started":1}
I20260927 15:16:08.807415 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:08.787184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35377 (local address 127.10.33.66:43737)
0927 15:16:08.787968 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:08.788010 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:08.789715 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:08.790441 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:08.790449 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:08.790866 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:08.791429 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:08.791446 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:08.792817 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:08.792823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:08.794916 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:08.794932 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:08.795866 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:08.795874 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:08.796154 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:08.796939 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:08.796975 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:08.799583 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:08.802745 (+ 3162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:08.802754 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:08.802774 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:08.803370 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:08.803824 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:08.803829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:08.803833 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:08.804117 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:08.804739 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:08.804747 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:08.805209 (+ 462us) client_negotiation.cc:769] Sending connection context
0927 15:16:08.805512 (+ 303us) client_negotiation.cc:240] Negotiation successful
0927 15:16:08.806077 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":324,"threads_started":1}
I20260927 15:16:08.808840 23082 heartbeater.cc:347] Connected to a master server at 127.10.33.126:35377
I20260927 15:16:08.809173 23082 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:08.810035 23082 heartbeater.cc:511] Master 127.10.33.126:35377 requested a full tablet report, sending...
I20260927 15:16:08.811820 22726 ts_manager.cc:194] Registered new tserver with Master: 1169cea5eb7b4fa2a4256c321ab5e016 (127.10.33.66:39089)
I20260927 15:16:08.812940 22726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:43737
WARNING: no policy specified for kudu/127.10.33.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
I20260927 15:16:08.845485 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.33.84:57123/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:35377
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:16:09.086632 22939 heartbeater.cc:503] Master 127.10.33.126:35377 was elected leader, sending a full tablet report...
W20260927 15:16:09.170349 23088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:09.170639 23088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:09.170761 23088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:09.178022 23088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:09.178157 23088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:09.178316 23088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:16:09.188153 23088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.33.84:57123/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:35377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:09.189397 23088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:09.190660 23088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:09.203687 23093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:09.203706 23094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:09.203737 23096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:09.204618 23088 server_base.cc:1034] running on GCE node
I20260927 15:16:09.205204 23088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:09.206252 23088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:09.207453 23088 hybrid_clock.cc:648] HybridClock initialized: now 1790522169207389 us; error 48 us; skew 500 ppm
I20260927 15:16:09.207755 23088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:09 dist-test-slave-f5h6 krb5kdc[22250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522169, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:09.210933 23088 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:16:09.212392 23088 webserver.cc:460] Webserver started at http://127.10.33.67:35979/ using document root <none> and password file <none>
I20260927 15:16:09.213091 23088 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:09.213207 23088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:09.213490 23088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:16:09 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522169, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
I20260927 15:16:09.229288 23088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "848e5a0eb3cb4e3b9f210337d03343ba"
format_stamp: "Formatted at 2026-09-27 15:16:09 on dist-test-slave-f5h6"
server_key: "fdf398393cff2efdfae4b9cf3490583c"
server_key_iv: "3539aa0d4f0f783f6fe6d4db1ef875ed"
server_key_version: "kuduclusterkey@0"
I20260927 15:16:09.230270 23088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "848e5a0eb3cb4e3b9f210337d03343ba"
format_stamp: "Formatted at 2026-09-27 15:16:09 on dist-test-slave-f5h6"
server_key: "fdf398393cff2efdfae4b9cf3490583c"
server_key_iv: "3539aa0d4f0f783f6fe6d4db1ef875ed"
server_key_version: "kuduclusterkey@0"
I20260927 15:16:09.235879 23088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:16:09.253330 23103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:09.255103 23088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:16:09.255303 23088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "848e5a0eb3cb4e3b9f210337d03343ba"
format_stamp: "Formatted at 2026-09-27 15:16:09 on dist-test-slave-f5h6"
server_key: "fdf398393cff2efdfae4b9cf3490583c"
server_key_iv: "3539aa0d4f0f783f6fe6d4db1ef875ed"
server_key_version: "kuduclusterkey@0"
I20260927 15:16:09.255559 23088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:09.283424 23088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:09.288487 23088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:09.288805 23088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:09.289997 23088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:09.291671 23088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:09.291764 23088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:09.291880 23088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:09.291946 23088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:09.322710 23088 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:44381
I20260927 15:16:09.322777 23215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:44381 every 8 connection(s)
I20260927 15:16:09.324034 23088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:16:09.328809 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 23088
I20260927 15:16:09.328984 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:16:09.329547 10373 external_mini_cluster.cc:1428] Setting key d7d9b21316d504d7d0ce93e51eba7216
Sep 27 15:16:09 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522169, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:16:09.343969 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:09.326394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:41453 (local address 127.10.33.126:35377)
0927 15:16:09.326849 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:09.326853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:09.327637 (+ 784us) server_negotiation.cc:407] Connection header received
0927 15:16:09.329841 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:09.329846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:09.329977 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:09.330084 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:09.331834 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:09.332603 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:09.334278 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:09.334695 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:09.334807 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:09.338073 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:09.338078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:09.338081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:09.338114 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:09.340673 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:09.341411 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:09.341417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:09.341419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:09.341557 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:09.342128 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:09.342132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:09.342134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:09.342410 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:09.342564 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:09.343344 (+ 780us) server_negotiation.cc:299] Negotiation successful
0927 15:16:09.343594 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":225,"threads_started":1}
I20260927 15:16:09.345243 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:09.326786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35377 (local address 127.10.33.67:41453)
0927 15:16:09.327476 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:09.327519 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:09.329500 (+ 1981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:09.330377 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:09.330386 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:09.330909 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:09.331652 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:09.331674 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:09.332779 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:09.332783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:09.334140 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:09.334153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:09.334931 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:09.334938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:09.335166 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:09.335804 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:09.335828 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:09.337813 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:09.340819 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:09.340829 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:09.340849 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:09.341292 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:09.341664 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:09.341669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:09.341673 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:09.342007 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:09.342536 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:09.342545 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:09.342905 (+ 360us) client_negotiation.cc:769] Sending connection context
0927 15:16:09.343211 (+ 306us) client_negotiation.cc:240] Negotiation successful
0927 15:16:09.343765 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":288,"threads_started":1}
I20260927 15:16:09.346747 23216 heartbeater.cc:347] Connected to a master server at 127.10.33.126:35377
I20260927 15:16:09.347049 23216 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:09.347744 23216 heartbeater.cc:511] Master 127.10.33.126:35377 requested a full tablet report, sending...
I20260927 15:16:09.349144 22726 ts_manager.cc:194] Registered new tserver with Master: 848e5a0eb3cb4e3b9f210337d03343ba (127.10.33.67:44381)
I20260927 15:16:09.349995 22726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:41453
I20260927 15:16:09.358443 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:16:09.360001 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 22806
Sep 27 15:16:09 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.65@KRBTEST.COM
I20260927 15:16:09.376951 23221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:09.360901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43572 (local address 127.10.33.65:39617)
0927 15:16:09.361505 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:09.361514 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:09.361547 (+ 33us) server_negotiation.cc:407] Connection header received
0927 15:16:09.361672 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:09.361682 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:09.361988 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:09.362128 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:09.363563 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:09.364549 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:09.366129 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:09.366777 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:09.366913 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:09.371341 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:09.371352 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:09.371369 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:09.371408 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:09.374096 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:09.374883 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:09.374890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:09.374892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:09.375050 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:09.375404 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:09.375408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:09.375410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:09.375747 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:09.376127 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:09.376336 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 15:16:09.376607 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":266,"threads_started":1}
I20260927 15:16:09.378284 22934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:09.465251 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 22806
I20260927 15:16:09.477725 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 22945
Sep 27 15:16:09 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.66@KRBTEST.COM
I20260927 15:16:09.497749 23223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:09.478685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46290 (local address 127.10.33.66:39089)
0927 15:16:09.479234 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:09.479241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:09.479264 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:16:09.479322 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:09.479329 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:09.479544 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:09.479668 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:09.480855 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:09.481816 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:09.483197 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:09.483721 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:09.483814 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:09.492070 (+ 8256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:09.492085 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:09.492101 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:09.492145 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:09.495055 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:09.495875 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:09.495883 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:09.495886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:09.496044 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:09.496338 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:09.496342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:09.496344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:09.496718 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:09.496994 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:09.497179 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 15:16:09.497388 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":303,"threads_started":1}
I20260927 15:16:09.499101 23077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:09.580818 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 22945
I20260927 15:16:09.593014 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 23088
Sep 27 15:16:09 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.67@KRBTEST.COM
I20260927 15:16:09.609970 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:09.594030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47348 (local address 127.10.33.67:44381)
0927 15:16:09.594751 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:09.594758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:09.594787 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:16:09.594843 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:09.594851 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:09.595100 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:09.595253 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:09.596606 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:09.597601 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:09.598941 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:09.599507 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:09.599619 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:09.604058 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:09.604077 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:09.604093 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:09.604136 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:09.607114 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:09.607875 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:09.607882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:09.607885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:09.608020 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:09.608374 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:09.608378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:09.608380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:09.608735 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:09.609038 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:09.609342 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 15:16:09.609605 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":418,"threads_started":1}
I20260927 15:16:09.611225 23211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:09.696134 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 23088
I20260927 15:16:09.708691 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 22694
I20260927 15:16:09.709951 22756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:09.813993 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 22694
I20260927 15:16:09.832855 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:35377
--webserver_interface=127.10.33.126
--webserver_port=40363
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:35377
--ranger_config_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.33.84:57123/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.33.84:57123/kms with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:10.155583 23228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:10.155905 23228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:10.155995 23228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:10.163465 23228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:10.163643 23228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:10.163691 23228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:10.163749 23228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:10.163795 23228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:16:10.174608 23228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.33.84:57123/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:35377
--ranger_config_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:35377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=40363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:10.175890 23228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:10.177141 23228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:10.189193 23233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:10.189154 23236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:10.189235 23234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:10.190407 23228 server_base.cc:1034] running on GCE node
I20260927 15:16:10.191216 23228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:10.192704 23228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:10.193950 23228 hybrid_clock.cc:648] HybridClock initialized: now 1790522170193885 us; error 55 us; skew 500 ppm
I20260927 15:16:10.194308 23228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:10 dist-test-slave-f5h6 krb5kdc[22250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522170, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:10.197428 23228 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:16:10.198988 23228 webserver.cc:460] Webserver started at http://127.10.33.126:40363/ using document root <none> and password file <none>
I20260927 15:16:10.199663 23228 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:10.199782 23228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:16:10 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522170, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
I20260927 15:16:10.261771 23228 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 15:16:10.265110 23244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:10.266631 23228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:16:10.266806 23228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "d2baac03b77044029c44e1dd64283ff5"
format_stamp: "Formatted at 2026-09-27 15:16:06 on dist-test-slave-f5h6"
server_key: "360b641bda968d47e18c6f88a0f11e98"
server_key_iv: "8cbdd550b525666689079d32ffb26df4"
server_key_version: "kuduclusterkey@0"
I20260927 15:16:10.267767 23228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:10.286486 23228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:10.290645 23228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:10.290921 23228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:10.308580 23228 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:35377
I20260927 15:16:10.308600 23295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:35377 every 8 connection(s)
I20260927 15:16:10.310240 23228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:16:10.315480 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 23228
I20260927 15:16:10.316270 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.33.84:57123/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:39617
--local_ip_for_outbound_sockets=127.10.33.65
--tserver_master_addrs=127.10.33.126:35377
--webserver_port=35073
--webserver_interface=127.10.33.65
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.33.84:57123/kms with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:16:10.317090 23296 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:16:10.320390 23296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5: Bootstrap starting.
I20260927 15:16:10.339352 23296 log.cc:824] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:10.348371 23296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:16:10.348883 23296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5: Bootstrap complete.
I20260927 15:16:10.358376 23296 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2baac03b77044029c44e1dd64283ff5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35377 } }
I20260927 15:16:10.359241 23296 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2baac03b77044029c44e1dd64283ff5, State: Initialized, Role: FOLLOWER
I20260927 15:16:10.359923 23296 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2baac03b77044029c44e1dd64283ff5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35377 } }
I20260927 15:16:10.360147 23296 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:16:10.360260 23296 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:16:10.360424 23296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:16:10.364344 23296 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2baac03b77044029c44e1dd64283ff5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35377 } }
I20260927 15:16:10.364871 23296 leader_election.cc:302] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2baac03b77044029c44e1dd64283ff5; no voters:
I20260927 15:16:10.365456 23296 leader_election.cc:288] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:16:10.365603 23300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:16:10.366673 23300 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [term 2 LEADER]: Becoming Leader. State: Replica: d2baac03b77044029c44e1dd64283ff5, State: Running, Role: LEADER
I20260927 15:16:10.367304 23300 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2baac03b77044029c44e1dd64283ff5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35377 } }
I20260927 15:16:10.367592 23296 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:10.370016 23301 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d2baac03b77044029c44e1dd64283ff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2baac03b77044029c44e1dd64283ff5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35377 } } }
I20260927 15:16:10.370307 23301 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:10.369669 23302 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2baac03b77044029c44e1dd64283ff5. Latest consensus state: current_term: 2 leader_uuid: "d2baac03b77044029c44e1dd64283ff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2baac03b77044029c44e1dd64283ff5" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35377 } } }
I20260927 15:16:10.370895 23302 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:10.379396 23306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:16:10.385447 23306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:16:10.386987 23306 catalog_manager.cc:1259] Loaded cluster ID: 9d6ff55959f1488fbfd8315cab6453b1
I20260927 15:16:10.387192 23306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:16:10.391403 23306 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:16:10.393040 23306 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5: Loaded TSK: 0
I20260927 15:16:10.394809 23306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:16:10.397470 23296 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260927 15:16:10.761000 23298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:10.761469 23298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:10.761687 23298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:10.773578 23298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:10.773825 23298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:10.774093 23298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:16:10.789777 23298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.33.84:57123/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:39617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=35073
--tserver_master_addrs=127.10.33.126:35377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:10.791891 23298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:10.793671 23298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:10.819433 23334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:10.819911 23333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:10.822072 23336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:10.823364 23298 server_base.cc:1034] running on GCE node
I20260927 15:16:10.824329 23298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:10.826133 23298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:10.827455 23298 hybrid_clock.cc:648] HybridClock initialized: now 1790522170827378 us; error 61 us; skew 500 ppm
I20260927 15:16:10.827850 23298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:10 dist-test-slave-f5h6 krb5kdc[22250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522170, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:10.832033 23298 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:16:10.834764 23298 webserver.cc:460] Webserver started at http://127.10.33.65:35073/ using document root <none> and password file <none>
I20260927 15:16:10.835861 23298 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:10.836089 23298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:16:10 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522170, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
I20260927 15:16:10.870854 23298 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.000s sys 0.004s
I20260927 15:16:10.879137 23344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:10.881186 23298 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.003s
I20260927 15:16:10.881493 23298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "dc57b1c5139f42249a8d061b9b3ade06"
format_stamp: "Formatted at 2026-09-27 15:16:07 on dist-test-slave-f5h6"
server_key: "102e4f3936b95d95a8c984533454ba50"
server_key_iv: "31193d4391def16cd36bcd201c67d28f"
server_key_version: "kuduclusterkey@0"
I20260927 15:16:10.882910 23298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:10.952513 23298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:10.957108 23298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:10.957427 23298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:10.959306 23298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:10.961638 23298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:10.961841 23298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 15:16:10.962071 23298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:10.962212 23298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:11.021787 23298 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:39617
I20260927 15:16:11.021835 23457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:39617 every 8 connection(s)
I20260927 15:16:11.023496 23298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:16:11.033228 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 23298
I20260927 15:16:11.034083 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.33.84:57123/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:39089
--local_ip_for_outbound_sockets=127.10.33.66
--tserver_master_addrs=127.10.33.126:35377
--webserver_port=43835
--webserver_interface=127.10.33.66
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.33.84:57123/kms with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:16:11 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522170, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:16:11.066923 23461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:11.027115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35377 (local address 127.10.33.65:49901)
0927 15:16:11.031596 (+ 4481us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:11.031648 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:11.033666 (+ 2018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:11.035913 (+ 2247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:11.035924 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:11.036549 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:11.037238 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:11.037269 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:11.039295 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:11.039300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:11.040968 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:11.040983 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:11.042811 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:11.042818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:11.043150 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:11.046099 (+ 2949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:11.046138 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:11.054887 (+ 8749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:11.060148 (+ 5261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:11.060160 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:11.060190 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:11.061255 (+ 1065us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:11.062140 (+ 885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:11.062148 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:11.062152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:11.062701 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:11.063939 (+ 1238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:11.063949 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:11.064480 (+ 531us) client_negotiation.cc:769] Sending connection context
0927 15:16:11.064861 (+ 381us) client_negotiation.cc:240] Negotiation successful
0927 15:16:11.065492 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4150,"thread_start_us":3913,"threads_started":1}
I20260927 15:16:11.067214 23460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:11.026972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:49901 (local address 127.10.33.126:35377)
0927 15:16:11.032565 (+ 5593us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:11.032584 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:11.032668 (+ 84us) server_negotiation.cc:407] Connection header received
0927 15:16:11.034387 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:11.034405 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:11.034812 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:11.035176 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:11.037506 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:11.039127 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:11.041157 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:11.042203 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:11.042391 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:11.055320 (+ 12929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:11.055333 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:11.055356 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:11.055419 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:11.059895 (+ 4476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:11.061504 (+ 1609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:11.061516 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:11.061526 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:11.061798 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:11.062891 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:11.062898 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:11.062902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:11.063677 (+ 775us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:11.063930 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:11.065141 (+ 1211us) server_negotiation.cc:299] Negotiation successful
0927 15:16:11.065659 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5319,"thread_start_us":5072,"threads_started":1}
I20260927 15:16:11.071379 23458 heartbeater.cc:347] Connected to a master server at 127.10.33.126:35377
I20260927 15:16:11.071715 23458 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:11.072676 23458 heartbeater.cc:511] Master 127.10.33.126:35377 requested a full tablet report, sending...
W20260927 15:16:11.076251 23458 heartbeater.cc:644] Failed to heartbeat to 127.10.33.126:35377 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 15:16:11.076486 23458 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:11.076769 23458 heartbeater.cc:511] Master 127.10.33.126:35377 requested a full tablet report, sending...
I20260927 15:16:11.078369 23458 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:11.078753 23458 heartbeater.cc:511] Master 127.10.33.126:35377 requested a full tablet report, sending...
W20260927 15:16:11.080060 23458 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:16:11.461982 23462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:11.462544 23462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:11.462772 23462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:11.474984 23462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:11.475208 23462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:11.475440 23462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:16:11.489207 23462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.33.84:57123/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:39089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=43835
--tserver_master_addrs=127.10.33.126:35377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:11.490814 23462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:11.492658 23462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:11.522115 23477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:11.523308 23480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:11.523931 23462 server_base.cc:1034] running on GCE node
W20260927 15:16:11.523329 23476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:11.524996 23462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:11.526404 23462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:11.527643 23462 hybrid_clock.cc:648] HybridClock initialized: now 1790522171527566 us; error 72 us; skew 500 ppm
I20260927 15:16:11.528044 23462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:11 dist-test-slave-f5h6 krb5kdc[22250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522171, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:11.536113 23462 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:16:11.550307 23462 webserver.cc:460] Webserver started at http://127.10.33.66:43835/ using document root <none> and password file <none>
I20260927 15:16:11.551432 23462 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:11.551628 23462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:16:11 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522171, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
I20260927 15:16:11.579957 23462 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.000s sys 0.006s
I20260927 15:16:11.583890 23489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:11.585690 23462 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 15:16:11.585981 23462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "1169cea5eb7b4fa2a4256c321ab5e016"
format_stamp: "Formatted at 2026-09-27 15:16:08 on dist-test-slave-f5h6"
server_key: "b6a577e3d0867299df1bbfc8caf9efa7"
server_key_iv: "edab07d4b722eb862b2d81642eed0aa2"
server_key_version: "kuduclusterkey@0"
I20260927 15:16:11.587435 23462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 27 15:16:11 dist-test-slave-f5h6 krb5kdc[22250](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790522171, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:11.629652 23462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:11.639009 23462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:11.639405 23462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:11.641741 23462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:11.643867 23462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:11.644001 23462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:11.644143 23462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:11.644255 23462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 15:16:11.710675 23462 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:39089
I20260927 15:16:11.710696 23602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:39089 every 8 connection(s)
I20260927 15:16:11.712561 23462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:16:11.722388 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 23462
I20260927 15:16:11.723191 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.33.84:57123/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:44381
--local_ip_for_outbound_sockets=127.10.33.67
--tserver_master_addrs=127.10.33.126:35377
--webserver_port=35979
--webserver_interface=127.10.33.67
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.33.84:57123/kms with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:16:11 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522171, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:16:11.748358 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:11.715946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35377 (local address 127.10.33.66:47187)
0927 15:16:11.726649 (+ 10703us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:11.726713 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:11.728658 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:11.729464 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:11.729473 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:11.729999 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:11.730638 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:11.730658 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:11.731892 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:11.731897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:11.733848 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:11.733897 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:11.735017 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:11.735033 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:11.735323 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:11.736182 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:11.736212 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:11.738805 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:11.742881 (+ 4076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:11.742892 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:11.742912 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:11.743592 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:11.744079 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:11.744084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:11.744088 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:11.744570 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:11.745337 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:11.745345 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:11.745890 (+ 545us) client_negotiation.cc:769] Sending connection context
0927 15:16:11.746235 (+ 345us) client_negotiation.cc:240] Negotiation successful
0927 15:16:11.746865 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10380,"thread_start_us":10138,"threads_started":1}
I20260927 15:16:11.749529 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:11.716509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:47187 (local address 127.10.33.126:35377)
0927 15:16:11.726022 (+ 9513us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:11.726029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:11.727257 (+ 1228us) server_negotiation.cc:407] Connection header received
0927 15:16:11.728983 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:11.728989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:11.729086 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:11.729181 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:11.730832 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:11.731682 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:11.734060 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:11.734745 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:11.734913 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:11.739108 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:11.739116 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:11.739121 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:11.739167 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:11.742680 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:11.743762 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:11.743768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:11.743772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:11.743931 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:11.744724 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:11.744730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:11.744733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:11.745119 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:11.745261 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:11.748892 (+ 3631us) server_negotiation.cc:299] Negotiation successful
0927 15:16:11.749160 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9321,"thread_start_us":9192,"threads_started":1}
I20260927 15:16:11.752161 23603 heartbeater.cc:347] Connected to a master server at 127.10.33.126:35377
I20260927 15:16:11.752514 23603 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:11.753360 23603 heartbeater.cc:511] Master 127.10.33.126:35377 requested a full tablet report, sending...
I20260927 15:16:11.755695 23261 ts_manager.cc:194] Registered new tserver with Master: 1169cea5eb7b4fa2a4256c321ab5e016 (127.10.33.66:39089)
I20260927 15:16:11.757995 23261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:47187
Sep 27 15:16:12 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522171, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
I20260927 15:16:12.080197 23458 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:12.080675 23458 heartbeater.cc:511] Master 127.10.33.126:35377 requested a full tablet report, sending...
I20260927 15:16:12.082041 23261 ts_manager.cc:194] Registered new tserver with Master: dc57b1c5139f42249a8d061b9b3ade06 (127.10.33.65:39617)
I20260927 15:16:12.083200 23261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:49901
W20260927 15:16:12.139761 23607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:12.140082 23607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:12.140249 23607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:12.150684 23607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:12.150876 23607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:12.151073 23607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:16:12.166824 23607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.33.84:57123/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:44381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=35979
--tserver_master_addrs=127.10.33.126:35377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:12.168619 23607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:12.170305 23607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:12.188365 23613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:12.188926 23612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:12.189023 23607 server_base.cc:1034] running on GCE node
W20260927 15:16:12.189374 23615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:12.190068 23607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:12.191115 23607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:12.192375 23607 hybrid_clock.cc:648] HybridClock initialized: now 1790522172192313 us; error 56 us; skew 500 ppm
I20260927 15:16:12.192684 23607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:12 dist-test-slave-f5h6 krb5kdc[22250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522172, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:12.195924 23607 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:16:12.199414 23607 webserver.cc:460] Webserver started at http://127.10.33.67:35979/ using document root <none> and password file <none>
I20260927 15:16:12.200100 23607 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:12.200237 23607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:16:12 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522172, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
I20260927 15:16:12.222290 23607 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:16:12.225209 23623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:12.226766 23607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:16:12.226938 23607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "848e5a0eb3cb4e3b9f210337d03343ba"
format_stamp: "Formatted at 2026-09-27 15:16:09 on dist-test-slave-f5h6"
server_key: "fdf398393cff2efdfae4b9cf3490583c"
server_key_iv: "3539aa0d4f0f783f6fe6d4db1ef875ed"
server_key_version: "kuduclusterkey@0"
I20260927 15:16:12.227945 23607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:12.269374 23607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:12.273726 23607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:12.274044 23607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:12.275116 23607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:12.276697 23607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:12.276872 23607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:12.277038 23607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:12.277153 23607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:12.356002 23607 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:44381
I20260927 15:16:12.358143 23607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:16:12.359674 23736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:44381 every 8 connection(s)
I20260927 15:16:12.360535 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 23607
Sep 27 15:16:12 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522171, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for HTTP/127.10.33.84@KRBTEST.COM
Sep 27 15:16:12 dist-test-slave-f5h6 krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522172, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:16:12.448089 23741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:12.363228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47124 (local address 127.10.33.126:35377)
0927 15:16:12.404998 (+ 41770us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:12.405005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:12.410682 (+ 5677us) server_negotiation.cc:407] Connection header received
0927 15:16:12.410767 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:12.410773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:12.410886 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:12.410995 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:12.413971 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.415128 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.421986 (+ 6858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.422598 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.422699 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:12.437739 (+ 15040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:12.437747 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:12.437753 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:12.437805 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:12.441453 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:12.444203 (+ 2750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:12.444209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:12.444213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:12.444465 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:12.445144 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:12.445150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:12.445154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:12.445555 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:12.445799 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:12.447447 (+ 1648us) server_negotiation.cc:299] Negotiation successful
0927 15:16:12.447709 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41616,"thread_start_us":52649,"threads_started":1}
I20260927 15:16:12.459765 23742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:12.371424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35377 (local address 127.10.33.67:42397)
0927 15:16:12.405597 (+ 34173us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:12.405655 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:12.408433 (+ 2778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:12.418059 (+ 9626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:12.418073 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:12.418820 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:12.419778 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:12.419825 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.421521 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.421527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:12.429805 (+ 8278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:12.429834 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.435385 (+ 5551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.435396 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:12.435748 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:12.442811 (+ 7063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:12.442856 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:12.447009 (+ 4153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:12.454647 (+ 7638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:12.454657 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:12.454678 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:12.455326 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:12.455791 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:12.455797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:12.455800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:12.456095 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:12.456827 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:12.456836 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:12.457307 (+ 471us) client_negotiation.cc:769] Sending connection context
0927 15:16:12.457637 (+ 330us) client_negotiation.cc:240] Negotiation successful
0927 15:16:12.458267 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33810,"thread_start_us":37445,"threads_started":1}
I20260927 15:16:12.460987 23740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:12.370648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:42397 (local address 127.10.33.126:35377)
0927 15:16:12.413748 (+ 43100us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:12.413754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:12.413774 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:16:12.413828 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:12.413834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:12.417290 (+ 3456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:12.417411 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:12.420068 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.421301 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.430115 (+ 8814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.433173 (+ 3058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.433336 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:12.449563 (+ 16227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:12.449570 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:12.449575 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:12.449616 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:12.454398 (+ 4782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:12.455498 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:12.455504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:12.455508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:12.455664 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:12.456233 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:12.456238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:12.456241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:12.456609 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:12.456750 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:12.460357 (+ 3607us) server_negotiation.cc:299] Negotiation successful
0927 15:16:12.460622 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42953,"thread_start_us":38224,"threads_started":1}
I20260927 15:16:12.463897 23737 heartbeater.cc:347] Connected to a master server at 127.10.33.126:35377
I20260927 15:16:12.464232 23737 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:12.465135 23737 heartbeater.cc:511] Master 127.10.33.126:35377 requested a full tablet report, sending...
I20260927 15:16:12.467756 23260 ts_manager.cc:194] Registered new tserver with Master: 848e5a0eb3cb4e3b9f210337d03343ba (127.10.33.67:44381)
I20260927 15:16:12.469096 23260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:42397
I20260927 15:16:12.482688 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:16:12.556389 23740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:12.527510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47126 (local address 127.10.33.126:35377)
0927 15:16:12.527708 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:12.527716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:12.538534 (+ 10818us) server_negotiation.cc:407] Connection header received
0927 15:16:12.538839 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:12.538846 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:12.538936 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:12.539065 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:12.541300 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.542416 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.544416 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.545046 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.545157 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:12.549285 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:12.549292 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:12.549297 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:12.549344 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:12.552885 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:12.553964 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:12.553971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:12.553975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:12.554126 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:12.554584 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:12.554590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:12.554593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:12.554975 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:12.555193 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:12.555783 (+ 590us) server_negotiation.cc:299] Negotiation successful
0927 15:16:12.556049 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:16:12.563508 23260 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:16:12.575624 23260 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:16:12.667677 23755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:12.604184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47356 (local address 127.10.33.67:44381)
0927 15:16:12.652837 (+ 48653us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:12.652847 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:12.654541 (+ 1694us) server_negotiation.cc:407] Connection header received
0927 15:16:12.654673 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:12.654682 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:12.654991 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:12.655221 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:12.656486 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.658580 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.663258 (+ 4678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.664939 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.665353 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:12.665724 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:12.667074 (+ 1350us) server_negotiation.cc:299] Negotiation successful
0927 15:16:12.667283 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48480,"thread_start_us":48299,"threads_started":1}
I20260927 15:16:12.668242 23754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:12.603477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:44381 (local address 127.0.0.1:47356)
0927 15:16:12.654299 (+ 50822us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:12.654321 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:12.654424 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:12.655631 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:12.655640 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:12.655682 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:12.656228 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:12.656244 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.658852 (+ 2608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.658859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:12.662005 (+ 3146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:12.662032 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.666280 (+ 4248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.666290 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:12.666631 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:12.666703 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 15:16:12.666874 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 15:16:12.667463 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50262,"thread_start_us":50087,"threads_started":1}
I20260927 15:16:12.673914 23671 tablet_service.cc:1467] Processing CreateTablet for tablet 99d829b5ecc4489ca38a6c60cd040a07 (DEFAULT_TABLE table=test-table [id=b3eb5c4075e0493191c5d0b9c9e9c2b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:12.675486 23671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d829b5ecc4489ca38a6c60cd040a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:12.678188 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:12.617894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46292 (local address 127.10.33.66:39089)
0927 15:16:12.651017 (+ 33123us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:12.651028 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:12.666718 (+ 15690us) server_negotiation.cc:407] Connection header received
0927 15:16:12.668182 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:12.668193 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:12.668536 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:12.669217 (+ 681us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:12.670419 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.672232 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.674866 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.676334 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.676534 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:12.676743 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:12.677496 (+ 753us) server_negotiation.cc:299] Negotiation successful
0927 15:16:12.677783 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32938,"thread_start_us":32806,"threads_started":1}
I20260927 15:16:12.679147 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:12.617227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:39089 (local address 127.0.0.1:46292)
0927 15:16:12.666559 (+ 49332us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:12.666584 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:12.666853 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:12.669721 (+ 2868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:12.669726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:12.669762 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:12.670189 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:12.670205 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.672470 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.672476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:12.674645 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:12.674663 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.676959 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.676967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:12.677223 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:12.677270 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 15:16:12.678504 (+ 1234us) client_negotiation.cc:240] Negotiation successful
0927 15:16:12.678660 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49092,"mutex_wait_us":5,"thread_start_us":32955,"threads_started":1}
I20260927 15:16:12.679561 23756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:12.599127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:39617 (local address 127.0.0.1:43586)
0927 15:16:12.651959 (+ 52832us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:12.652007 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:12.654072 (+ 2065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:12.663537 (+ 9465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:12.663544 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:12.663581 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:12.664003 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:12.664017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.668206 (+ 4189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.668215 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:12.671003 (+ 2788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:12.671021 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.677008 (+ 5987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.677016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:12.677257 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:12.677295 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 15:16:12.677384 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 15:16:12.679154 (+ 1770us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51354,"mutex_wait_us":28,"thread_start_us":66907,"threads_started":1}
I20260927 15:16:12.680932 23753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:12.599452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43586 (local address 127.10.33.65:39617)
0927 15:16:12.662730 (+ 63278us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:12.662741 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:12.662778 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:16:12.662908 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:12.662918 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:12.663259 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:12.664257 (+ 998us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:12.665118 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.667942 (+ 2824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.674842 (+ 6900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.676712 (+ 1870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.680031 (+ 3319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:12.680309 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:12.680398 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 15:16:12.680596 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63052,"thread_start_us":63460,"threads_started":1}
I20260927 15:16:12.683609 23537 tablet_service.cc:1467] Processing CreateTablet for tablet 99d829b5ecc4489ca38a6c60cd040a07 (DEFAULT_TABLE table=test-table [id=b3eb5c4075e0493191c5d0b9c9e9c2b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:12.684425 23393 tablet_service.cc:1467] Processing CreateTablet for tablet 99d829b5ecc4489ca38a6c60cd040a07 (DEFAULT_TABLE table=test-table [id=b3eb5c4075e0493191c5d0b9c9e9c2b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:12.685143 23537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d829b5ecc4489ca38a6c60cd040a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:12.685556 23393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d829b5ecc4489ca38a6c60cd040a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:12.693480 23765 tablet_bootstrap.cc:492] T 99d829b5ecc4489ca38a6c60cd040a07 P 848e5a0eb3cb4e3b9f210337d03343ba: Bootstrap starting.
I20260927 15:16:12.697575 23765 tablet_bootstrap.cc:654] T 99d829b5ecc4489ca38a6c60cd040a07 P 848e5a0eb3cb4e3b9f210337d03343ba: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:12.698997 23765 log.cc:824] T 99d829b5ecc4489ca38a6c60cd040a07 P 848e5a0eb3cb4e3b9f210337d03343ba: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:12.699939 23767 tablet_bootstrap.cc:492] T 99d829b5ecc4489ca38a6c60cd040a07 P dc57b1c5139f42249a8d061b9b3ade06: Bootstrap starting.
I20260927 15:16:12.701135 23766 tablet_bootstrap.cc:492] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016: Bootstrap starting.
I20260927 15:16:12.704103 23767 tablet_bootstrap.cc:654] T 99d829b5ecc4489ca38a6c60cd040a07 P dc57b1c5139f42249a8d061b9b3ade06: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:12.704562 23766 tablet_bootstrap.cc:654] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:12.704869 23765 tablet_bootstrap.cc:492] T 99d829b5ecc4489ca38a6c60cd040a07 P 848e5a0eb3cb4e3b9f210337d03343ba: No bootstrap required, opened a new log
I20260927 15:16:12.705188 23765 ts_tablet_manager.cc:1397] T 99d829b5ecc4489ca38a6c60cd040a07 P 848e5a0eb3cb4e3b9f210337d03343ba: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260927 15:16:12.705199 23767 log.cc:824] T 99d829b5ecc4489ca38a6c60cd040a07 P dc57b1c5139f42249a8d061b9b3ade06: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:12.705772 23766 log.cc:824] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:12.708156 23767 tablet_bootstrap.cc:492] T 99d829b5ecc4489ca38a6c60cd040a07 P dc57b1c5139f42249a8d061b9b3ade06: No bootstrap required, opened a new log
I20260927 15:16:12.708418 23767 ts_tablet_manager.cc:1397] T 99d829b5ecc4489ca38a6c60cd040a07 P dc57b1c5139f42249a8d061b9b3ade06: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 15:16:12.708799 23766 tablet_bootstrap.cc:492] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016: No bootstrap required, opened a new log
I20260927 15:16:12.709074 23766 ts_tablet_manager.cc:1397] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 15:16:12.715468 23765 raft_consensus.cc:348] T 99d829b5ecc4489ca38a6c60cd040a07 P 848e5a0eb3cb4e3b9f210337d03343ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc57b1c5139f42249a8d061b9b3ade06" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 39617 } } peers { permanent_uuid: "848e5a0eb3cb4e3b9f210337d03343ba" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 44381 } } peers { permanent_uuid: "1169cea5eb7b4fa2a4256c321ab5e016" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 39089 } }
I20260927 15:16:12.716187 23765 raft_consensus.cc:374] T 99d829b5ecc4489ca38a6c60cd040a07 P 848e5a0eb3cb4e3b9f210337d03343ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:12.716341 23765 raft_consensus.cc:729] T 99d829b5ecc4489ca38a6c60cd040a07 P 848e5a0eb3cb4e3b9f210337d03343ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 848e5a0eb3cb4e3b9f210337d03343ba, State: Initialized, Role: FOLLOWER
I20260927 15:16:12.717180 23765 consensus_queue.cc:260] T 99d829b5ecc4489ca38a6c60cd040a07 P 848e5a0eb3cb4e3b9f210337d03343ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc57b1c5139f42249a8d061b9b3ade06" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 39617 } } peers { permanent_uuid: "848e5a0eb3cb4e3b9f210337d03343ba" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 44381 } } peers { permanent_uuid: "1169cea5eb7b4fa2a4256c321ab5e016" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 39089 } }
I20260927 15:16:12.717558 23767 raft_consensus.cc:348] T 99d829b5ecc4489ca38a6c60cd040a07 P dc57b1c5139f42249a8d061b9b3ade06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc57b1c5139f42249a8d061b9b3ade06" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 39617 } } peers { permanent_uuid: "848e5a0eb3cb4e3b9f210337d03343ba" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 44381 } } peers { permanent_uuid: "1169cea5eb7b4fa2a4256c321ab5e016" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 39089 } }
I20260927 15:16:12.718128 23767 raft_consensus.cc:374] T 99d829b5ecc4489ca38a6c60cd040a07 P dc57b1c5139f42249a8d061b9b3ade06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:12.718281 23767 raft_consensus.cc:729] T 99d829b5ecc4489ca38a6c60cd040a07 P dc57b1c5139f42249a8d061b9b3ade06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc57b1c5139f42249a8d061b9b3ade06, State: Initialized, Role: FOLLOWER
I20260927 15:16:12.719198 23767 consensus_queue.cc:260] T 99d829b5ecc4489ca38a6c60cd040a07 P dc57b1c5139f42249a8d061b9b3ade06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc57b1c5139f42249a8d061b9b3ade06" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 39617 } } peers { permanent_uuid: "848e5a0eb3cb4e3b9f210337d03343ba" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 44381 } } peers { permanent_uuid: "1169cea5eb7b4fa2a4256c321ab5e016" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 39089 } }
I20260927 15:16:12.721305 23766 raft_consensus.cc:348] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc57b1c5139f42249a8d061b9b3ade06" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 39617 } } peers { permanent_uuid: "848e5a0eb3cb4e3b9f210337d03343ba" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 44381 } } peers { permanent_uuid: "1169cea5eb7b4fa2a4256c321ab5e016" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 39089 } }
I20260927 15:16:12.723809 23766 raft_consensus.cc:374] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:12.723984 23766 raft_consensus.cc:729] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1169cea5eb7b4fa2a4256c321ab5e016, State: Initialized, Role: FOLLOWER
I20260927 15:16:12.724959 23766 consensus_queue.cc:260] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc57b1c5139f42249a8d061b9b3ade06" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 39617 } } peers { permanent_uuid: "848e5a0eb3cb4e3b9f210337d03343ba" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 44381 } } peers { permanent_uuid: "1169cea5eb7b4fa2a4256c321ab5e016" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 39089 } }
I20260927 15:16:12.727139 23737 heartbeater.cc:503] Master 127.10.33.126:35377 was elected leader, sending a full tablet report...
I20260927 15:16:12.727524 23765 ts_tablet_manager.cc:1428] T 99d829b5ecc4489ca38a6c60cd040a07 P 848e5a0eb3cb4e3b9f210337d03343ba: Time spent starting tablet: real 0.022s user 0.008s sys 0.015s
I20260927 15:16:12.728542 23603 heartbeater.cc:503] Master 127.10.33.126:35377 was elected leader, sending a full tablet report...
I20260927 15:16:12.731078 23766 ts_tablet_manager.cc:1428] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016: Time spent starting tablet: real 0.022s user 0.015s sys 0.001s
I20260927 15:16:12.732017 23458 heartbeater.cc:503] Master 127.10.33.126:35377 was elected leader, sending a full tablet report...
I20260927 15:16:12.732254 23767 ts_tablet_manager.cc:1428] T 99d829b5ecc4489ca38a6c60cd040a07 P dc57b1c5139f42249a8d061b9b3ade06: Time spent starting tablet: real 0.024s user 0.020s sys 0.004s
W20260927 15:16:12.782387 23459 tablet.cc:2369] T 99d829b5ecc4489ca38a6c60cd040a07 P dc57b1c5139f42249a8d061b9b3ade06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:16:12.820425 23773 raft_consensus.cc:482] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:16:12.820730 23773 raft_consensus.cc:504] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc57b1c5139f42249a8d061b9b3ade06" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 39617 } } peers { permanent_uuid: "848e5a0eb3cb4e3b9f210337d03343ba" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 44381 } } peers { permanent_uuid: "1169cea5eb7b4fa2a4256c321ab5e016" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 39089 } }
I20260927 15:16:12.822548 23773 leader_election.cc:288] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc57b1c5139f42249a8d061b9b3ade06 (127.10.33.65:39617), 848e5a0eb3cb4e3b9f210337d03343ba (127.10.33.67:44381)
I20260927 15:16:12.829408 23753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:12.823136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:51945 (local address 127.10.33.65:39617)
0927 15:16:12.823368 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:12.823376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:12.824087 (+ 711us) server_negotiation.cc:407] Connection header received
0927 15:16:12.824283 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:12.824289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:12.824350 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:12.824417 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:12.825023 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.825738 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.827641 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.828425 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.828580 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:12.828657 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:12.829051 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 15:16:12.829177 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 15:16:12.829406 23755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:12.823121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:40973 (local address 127.10.33.67:44381)
0927 15:16:12.823374 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:12.823380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:12.824144 (+ 764us) server_negotiation.cc:407] Connection header received
0927 15:16:12.824299 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:12.824302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:12.824360 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:12.824438 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:12.825091 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.825838 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.827678 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.828443 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.828602 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:12.828677 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:12.829033 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 15:16:12.829175 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 15:16:12.829603 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:12.823097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:44381 (local address 127.10.33.66:40973)
0927 15:16:12.824027 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:12.824047 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:12.824178 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:12.824464 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:12.824477 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:12.824498 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:12.824925 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:12.824934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.825982 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.825986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:12.827529 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:12.827541 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.828709 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.828714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:12.828846 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:12.828911 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 15:16:12.828989 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 15:16:12.829129 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":45,"thread_start_us":523,"threads_started":1}
I20260927 15:16:12.829654 23775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:12.822994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:39617 (local address 127.10.33.66:51945)
0927 15:16:12.823975 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:12.823992 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:12.824138 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:12.824455 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:12.824459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:12.824481 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:12.824877 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:12.824890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.825899 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.825903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:12.827476 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:12.827491 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.828667 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.828675 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:12.828835 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:12.828920 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 15:16:12.829004 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 15:16:12.829137 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"mutex_wait_us":68,"thread_start_us":675,"threads_started":1}
I20260927 15:16:12.830740 23691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d829b5ecc4489ca38a6c60cd040a07" candidate_uuid: "1169cea5eb7b4fa2a4256c321ab5e016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "848e5a0eb3cb4e3b9f210337d03343ba" is_pre_election: true
I20260927 15:16:12.830742 23413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d829b5ecc4489ca38a6c60cd040a07" candidate_uuid: "1169cea5eb7b4fa2a4256c321ab5e016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc57b1c5139f42249a8d061b9b3ade06" is_pre_election: true
I20260927 15:16:12.831322 23691 raft_consensus.cc:2454] T 99d829b5ecc4489ca38a6c60cd040a07 P 848e5a0eb3cb4e3b9f210337d03343ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1169cea5eb7b4fa2a4256c321ab5e016 in term 0.
I20260927 15:16:12.831322 23413 raft_consensus.cc:2454] T 99d829b5ecc4489ca38a6c60cd040a07 P dc57b1c5139f42249a8d061b9b3ade06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1169cea5eb7b4fa2a4256c321ab5e016 in term 0.
I20260927 15:16:12.832022 23493 leader_election.cc:302] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1169cea5eb7b4fa2a4256c321ab5e016, 848e5a0eb3cb4e3b9f210337d03343ba; no voters:
I20260927 15:16:12.832432 23773 raft_consensus.cc:2781] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:16:12.832616 23773 raft_consensus.cc:482] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:16:12.832726 23773 raft_consensus.cc:3037] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:12.835027 23773 raft_consensus.cc:504] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc57b1c5139f42249a8d061b9b3ade06" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 39617 } } peers { permanent_uuid: "848e5a0eb3cb4e3b9f210337d03343ba" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 44381 } } peers { permanent_uuid: "1169cea5eb7b4fa2a4256c321ab5e016" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 39089 } }
I20260927 15:16:12.835708 23773 leader_election.cc:288] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016 [CANDIDATE]: Term 1 election: Requested vote from peers dc57b1c5139f42249a8d061b9b3ade06 (127.10.33.65:39617), 848e5a0eb3cb4e3b9f210337d03343ba (127.10.33.67:44381)
I20260927 15:16:12.836158 23413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d829b5ecc4489ca38a6c60cd040a07" candidate_uuid: "1169cea5eb7b4fa2a4256c321ab5e016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc57b1c5139f42249a8d061b9b3ade06"
I20260927 15:16:12.836241 23691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d829b5ecc4489ca38a6c60cd040a07" candidate_uuid: "1169cea5eb7b4fa2a4256c321ab5e016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "848e5a0eb3cb4e3b9f210337d03343ba"
I20260927 15:16:12.836380 23413 raft_consensus.cc:3037] T 99d829b5ecc4489ca38a6c60cd040a07 P dc57b1c5139f42249a8d061b9b3ade06 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:12.836433 23691 raft_consensus.cc:3037] T 99d829b5ecc4489ca38a6c60cd040a07 P 848e5a0eb3cb4e3b9f210337d03343ba [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:12.839499 23413 raft_consensus.cc:2454] T 99d829b5ecc4489ca38a6c60cd040a07 P dc57b1c5139f42249a8d061b9b3ade06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1169cea5eb7b4fa2a4256c321ab5e016 in term 1.
I20260927 15:16:12.839493 23691 raft_consensus.cc:2454] T 99d829b5ecc4489ca38a6c60cd040a07 P 848e5a0eb3cb4e3b9f210337d03343ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1169cea5eb7b4fa2a4256c321ab5e016 in term 1.
I20260927 15:16:12.839926 23493 leader_election.cc:302] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1169cea5eb7b4fa2a4256c321ab5e016, 848e5a0eb3cb4e3b9f210337d03343ba, dc57b1c5139f42249a8d061b9b3ade06; no voters:
I20260927 15:16:12.840226 23773 raft_consensus.cc:2781] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:12.840663 23773 raft_consensus.cc:686] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016 [term 1 LEADER]: Becoming Leader. State: Replica: 1169cea5eb7b4fa2a4256c321ab5e016, State: Running, Role: LEADER
I20260927 15:16:12.841147 23773 consensus_queue.cc:237] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc57b1c5139f42249a8d061b9b3ade06" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 39617 } } peers { permanent_uuid: "848e5a0eb3cb4e3b9f210337d03343ba" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 44381 } } peers { permanent_uuid: "1169cea5eb7b4fa2a4256c321ab5e016" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 39089 } }
I20260927 15:16:12.846037 23259 catalog_manager.cc:5495] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1169cea5eb7b4fa2a4256c321ab5e016 (127.10.33.66). New cstate: current_term: 1 leader_uuid: "1169cea5eb7b4fa2a4256c321ab5e016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc57b1c5139f42249a8d061b9b3ade06" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "848e5a0eb3cb4e3b9f210337d03343ba" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 44381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1169cea5eb7b4fa2a4256c321ab5e016" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 39089 } health_report { overall_health: HEALTHY } } }
I20260927 15:16:12.906611 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:12.900428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46302 (local address 127.10.33.66:39089)
0927 15:16:12.900694 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:12.900699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:12.900732 (+ 33us) server_negotiation.cc:407] Connection header received
0927 15:16:12.900817 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:12.900821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:12.900897 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:12.901016 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:16:12.901498 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.902317 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.903945 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.904459 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.904576 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:12.905390 (+ 814us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:16:12.906036 (+ 646us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:16:12.906158 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:12.906301 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 15:16:12.906389 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260927 15:16:12.915592 23691 raft_consensus.cc:1261] T 99d829b5ecc4489ca38a6c60cd040a07 P 848e5a0eb3cb4e3b9f210337d03343ba [term 1 FOLLOWER]: Refusing update from remote peer 1169cea5eb7b4fa2a4256c321ab5e016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:12.915592 23413 raft_consensus.cc:1261] T 99d829b5ecc4489ca38a6c60cd040a07 P dc57b1c5139f42249a8d061b9b3ade06 [term 1 FOLLOWER]: Refusing update from remote peer 1169cea5eb7b4fa2a4256c321ab5e016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:12.916937 23776 consensus_queue.cc:1035] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc57b1c5139f42249a8d061b9b3ade06" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:12.917359 23773 consensus_queue.cc:1035] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "848e5a0eb3cb4e3b9f210337d03343ba" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 44381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:16:12.926959 23738 tablet.cc:2369] T 99d829b5ecc4489ca38a6c60cd040a07 P 848e5a0eb3cb4e3b9f210337d03343ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:16:12.929829 23779 mvcc.cc:205] Tried to move back new op lower bound from 7333978820249128960 to 7333978819961491456. Current Snapshot: MvccSnapshot[applied={T|T < 7333978820249128960}]
I20260927 15:16:12.932106 23781 mvcc.cc:205] Tried to move back new op lower bound from 7333978820249128960 to 7333978819961491456. Current Snapshot: MvccSnapshot[applied={T|T < 7333978820249128960}]
I20260927 15:16:12.943825 23260 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:12.944159 23260 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:12.948339 23260 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2baac03b77044029c44e1dd64283ff5: Sending DeleteTablet for 3 replicas of tablet 99d829b5ecc4489ca38a6c60cd040a07
I20260927 15:16:12.949579 23393 tablet_service.cc:1514] Processing DeleteTablet for tablet 99d829b5ecc4489ca38a6c60cd040a07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:12 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:43586
I20260927 15:16:12.950022 23537 tablet_service.cc:1514] Processing DeleteTablet for tablet 99d829b5ecc4489ca38a6c60cd040a07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:12 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:46292
I20260927 15:16:12.950346 23671 tablet_service.cc:1514] Processing DeleteTablet for tablet 99d829b5ecc4489ca38a6c60cd040a07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:12 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:47356
I20260927 15:16:12.953013 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 23298
I20260927 15:16:12.959903 23791 tablet_replica.cc:330] T 99d829b5ecc4489ca38a6c60cd040a07 P 848e5a0eb3cb4e3b9f210337d03343ba: stopping tablet replica
I20260927 15:16:12.960453 23791 raft_consensus.cc:2229] T 99d829b5ecc4489ca38a6c60cd040a07 P 848e5a0eb3cb4e3b9f210337d03343ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:16:12.961028 23791 raft_consensus.cc:2258] T 99d829b5ecc4489ca38a6c60cd040a07 P 848e5a0eb3cb4e3b9f210337d03343ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:12.964004 23789 tablet_replica.cc:330] T 99d829b5ecc4489ca38a6c60cd040a07 P dc57b1c5139f42249a8d061b9b3ade06: stopping tablet replica
I20260927 15:16:12.964550 23789 raft_consensus.cc:2229] T 99d829b5ecc4489ca38a6c60cd040a07 P dc57b1c5139f42249a8d061b9b3ade06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:16:12.965128 23789 raft_consensus.cc:2258] T 99d829b5ecc4489ca38a6c60cd040a07 P dc57b1c5139f42249a8d061b9b3ade06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:12.965708 23791 ts_tablet_manager.cc:1905] T 99d829b5ecc4489ca38a6c60cd040a07 P 848e5a0eb3cb4e3b9f210337d03343ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:12.967656 23789 ts_tablet_manager.cc:1905] T 99d829b5ecc4489ca38a6c60cd040a07 P dc57b1c5139f42249a8d061b9b3ade06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:12.971902 23790 tablet_replica.cc:330] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016: stopping tablet replica
I20260927 15:16:12.975358 23789 ts_tablet_manager.cc:1918] T 99d829b5ecc4489ca38a6c60cd040a07 P dc57b1c5139f42249a8d061b9b3ade06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:16:12.975658 23789 log.cc:1196] T 99d829b5ecc4489ca38a6c60cd040a07 P dc57b1c5139f42249a8d061b9b3ade06: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/99d829b5ecc4489ca38a6c60cd040a07
I20260927 15:16:12.976342 23789 ts_tablet_manager.cc:1939] T 99d829b5ecc4489ca38a6c60cd040a07 P dc57b1c5139f42249a8d061b9b3ade06: Deleting consensus metadata
I20260927 15:16:12.977546 23791 ts_tablet_manager.cc:1918] T 99d829b5ecc4489ca38a6c60cd040a07 P 848e5a0eb3cb4e3b9f210337d03343ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:16:12.977725 23791 log.cc:1196] T 99d829b5ecc4489ca38a6c60cd040a07 P 848e5a0eb3cb4e3b9f210337d03343ba: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/99d829b5ecc4489ca38a6c60cd040a07
I20260927 15:16:12.978293 23791 ts_tablet_manager.cc:1939] T 99d829b5ecc4489ca38a6c60cd040a07 P 848e5a0eb3cb4e3b9f210337d03343ba: Deleting consensus metadata
I20260927 15:16:12.978821 23246 catalog_manager.cc:4841] TS dc57b1c5139f42249a8d061b9b3ade06 (127.10.33.65:39617): tablet 99d829b5ecc4489ca38a6c60cd040a07 (table test-table [id=b3eb5c4075e0493191c5d0b9c9e9c2b6]) successfully deleted
I20260927 15:16:12.979604 23790 raft_consensus.cc:2229] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:16:12.980419 23790 raft_consensus.cc:2258] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:12.981827 23248 catalog_manager.cc:4841] TS 848e5a0eb3cb4e3b9f210337d03343ba (127.10.33.67:44381): tablet 99d829b5ecc4489ca38a6c60cd040a07 (table test-table [id=b3eb5c4075e0493191c5d0b9c9e9c2b6]) successfully deleted
I20260927 15:16:12.982705 23753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:12.954671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43596 (local address 127.10.33.65:39617)
0927 15:16:12.954857 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:12.954863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:12.963639 (+ 8776us) server_negotiation.cc:407] Connection header received
0927 15:16:12.964063 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:12.964069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:12.964157 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:12.964233 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:12.966286 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.967402 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.969653 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:12.970366 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:12.970470 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:12.974073 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:12.974090 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:12.974108 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:12.974167 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:12.978311 (+ 4144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:12.979407 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:12.979416 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:12.979420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:12.979591 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:12.980081 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:12.980086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:12.980090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:12.980728 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:12.981291 (+ 563us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:12.982081 (+ 790us) server_negotiation.cc:299] Negotiation successful
0927 15:16:12.982333 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:16:12.983959 23453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:12.985229 23790 ts_tablet_manager.cc:1905] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:12.992333 23790 ts_tablet_manager.cc:1918] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:16:12.992529 23790 log.cc:1196] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/99d829b5ecc4489ca38a6c60cd040a07
I20260927 15:16:12.993022 23790 ts_tablet_manager.cc:1939] T 99d829b5ecc4489ca38a6c60cd040a07 P 1169cea5eb7b4fa2a4256c321ab5e016: Deleting consensus metadata
I20260927 15:16:12.994799 23245 catalog_manager.cc:4841] TS 1169cea5eb7b4fa2a4256c321ab5e016 (127.10.33.66:39089): tablet 99d829b5ecc4489ca38a6c60cd040a07 (table test-table [id=b3eb5c4075e0493191c5d0b9c9e9c2b6]) successfully deleted
I20260927 15:16:13.096690 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 23298
I20260927 15:16:13.111861 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 23462
I20260927 15:16:13.128573 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:13.112602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46308 (local address 127.10.33.66:39089)
0927 15:16:13.112817 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:13.112822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:13.112846 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:16:13.113082 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:13.113093 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:13.113229 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:13.113347 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:13.114636 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:13.115817 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:13.117107 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:13.117777 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:13.118044 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:13.120502 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:13.120522 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:13.120544 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:13.120595 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:13.125131 (+ 4536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:13.126194 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:13.126204 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:13.126208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:13.126415 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:13.126798 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:13.126803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:13.126806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:13.127301 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:13.127632 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:13.127911 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 15:16:13.128188 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 15:16:13.129281 23598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:13.242130 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 23462
I20260927 15:16:13.256326 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 23607
I20260927 15:16:13.269454 23755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:13.257146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47364 (local address 127.10.33.67:44381)
0927 15:16:13.257344 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:13.257349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:13.257367 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:16:13.257544 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:13.257549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:13.257635 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:13.257721 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:13.259310 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:13.260087 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:13.261416 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:13.261906 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:13.262023 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:13.264315 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:13.264326 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:13.264342 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:13.264376 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:13.266931 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:13.267563 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:13.267569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:13.267571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:13.267698 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:13.268019 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:13.268023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:13.268025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:13.268466 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:13.268689 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:13.268968 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 15:16:13.269133 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:16:13.270179 23732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:13.372349 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 23607
I20260927 15:16:13.385327 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 23228
I20260927 15:16:13.386530 23291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:13.498000 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 23228
I20260927 15:16:13.515733 10373 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 15:16:13.852064 10373 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 15:16:13.852285 10373 mini_ranger.cc:66] Stopping Ranger...
I20260927 15:16:14.188271 10373 mini_ranger.cc:68] Stopped Ranger
2026-09-27 15:16:14.188 UTC [22280] LOG: received smart shutdown request
2026-09-27 15:16:14.191 UTC [22280] LOG: background worker "logical replication launcher" (PID 22289) exited with exit code 1
2026-09-27 15:16:14.191 UTC [22284] LOG: shutting down
2026-09-27 15:16:14.389 UTC [22280] LOG: database system is shut down
2026-09-27T15:16:14Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54445 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:16:14 dist-test-slave-f5h6 krb5kdc[23805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:16:14 dist-test-slave-f5h6 krb5kdc[23805](info): set up 2 sockets
Sep 27 15:16:14 dist-test-slave-f5h6 krb5kdc[23805](info): commencing operation
krb5kdc: starting...
W20260927 15:16:15.526201 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:16:15 dist-test-slave-f5h6 krb5kdc[23805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:16:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:16:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:16:15.758402 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:35267
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:42261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:35267,127.10.33.125:39861,127.10.33.124:41243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:16.090122 23818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:16.090425 23818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:16.090523 23818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:16.097823 23818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:16.097970 23818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:16.098021 23818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:16.098076 23818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:16.098119 23818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:16:16.108260 23818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:42261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:35267,127.10.33.125:39861,127.10.33.124:41243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:35267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:16.109740 23818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:16.111100 23818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:16.122478 23823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:16.122515 23824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:16.122591 23826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:16.123703 23818 server_base.cc:1034] running on GCE node
I20260927 15:16:16.124644 23818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:16.126230 23818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:16.127485 23818 hybrid_clock.cc:648] HybridClock initialized: now 1790522176127434 us; error 36 us; skew 500 ppm
I20260927 15:16:16.127867 23818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:16 dist-test-slave-f5h6 krb5kdc[23805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522176, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:16.131348 23818 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:16:16.133024 23818 webserver.cc:460] Webserver started at http://127.10.33.126:35985/ using document root <none> and password file <none>
I20260927 15:16:16.133736 23818 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:16.133883 23818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:16.134202 23818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:16.137392 23818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "7cb58db88c164300ac2e07b35594cad8"
format_stamp: "Formatted at 2026-09-27 15:16:16 on dist-test-slave-f5h6"
server_key: "587e3b859d33f972774f4d1b288e7a76"
server_key_iv: "276a091809b66dac192147e16836e192"
server_key_version: "encryptionkey@0"
I20260927 15:16:16.138304 23818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "7cb58db88c164300ac2e07b35594cad8"
format_stamp: "Formatted at 2026-09-27 15:16:16 on dist-test-slave-f5h6"
server_key: "587e3b859d33f972774f4d1b288e7a76"
server_key_iv: "276a091809b66dac192147e16836e192"
server_key_version: "encryptionkey@0"
I20260927 15:16:16.144214 23818 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 15:16:16.148303 23833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:16.149996 23818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:16:16.150205 23818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "7cb58db88c164300ac2e07b35594cad8"
format_stamp: "Formatted at 2026-09-27 15:16:16 on dist-test-slave-f5h6"
server_key: "587e3b859d33f972774f4d1b288e7a76"
server_key_iv: "276a091809b66dac192147e16836e192"
server_key_version: "encryptionkey@0"
I20260927 15:16:16.150413 23818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:16.198133 23818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:16.202576 23818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:16.202854 23818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:16.222206 23818 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:35267
I20260927 15:16:16.222246 23884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:35267 every 8 connection(s)
I20260927 15:16:16.223869 23818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:16:16.228061 23885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:16.231781 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 23818
I20260927 15:16:16.232018 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:16:16.232805 10373 external_mini_cluster.cc:1428] Setting key 725411afb719d3585d65673102a4505c
I20260927 15:16:16.236740 23885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35267 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:16:16 dist-test-slave-f5h6 krb5kdc[23805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522176, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.33.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.125@KRBTEST.COM" created.
I20260927 15:16:16.261453 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.240836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41500 (local address 127.10.33.126:35267)
0927 15:16:16.242783 (+ 1947us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:16.242792 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:16.242839 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:16:16.244132 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:16.244140 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:16.244440 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:16.244634 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:16.245836 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:16.247671 (+ 1835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:16.249290 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:16.250210 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:16.250393 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:16.254046 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:16.254060 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:16.254081 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:16.254126 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:16.257212 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:16.258007 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:16.258014 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:16.258016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:16.258162 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:16.258644 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:16.258647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:16.258649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:16.259168 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:16.259317 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:16.259957 (+ 640us) server_negotiation.cc:299] Negotiation successful
0927 15:16:16.260248 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1726,"spinlock_wait_cycles":23808,"thread_start_us":1448,"threads_started":1}
I20260927 15:16:16.261497 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.240836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35267 (local address 127.0.0.1:41500)
0927 15:16:16.241806 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.241890 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:16.243732 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:16.244679 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:16.244684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:16.245282 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:16.245636 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:16.245653 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:16.247931 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:16.247936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:16.249129 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:16.249142 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:16.250461 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:16.250468 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:16.250601 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:16.251327 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:16.251350 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:16.253727 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:16.257351 (+ 3624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:16.257360 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:16.257379 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:16.257882 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:16.258291 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:16.258295 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:16.258299 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:16.258547 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:16.259312 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:16.259319 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:16.259583 (+ 264us) client_negotiation.cc:769] Sending connection context
0927 15:16:16.259855 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 15:16:16.260411 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":54,"thread_start_us":408,"threads_started":1}
I20260927 15:16:16.264297 23885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:16:16.265312 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.264899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39861 (local address 127.0.0.1:54674)
0927 15:16:16.265092 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.265202 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260927 15:16:16.265527 23835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39861: connect: Connection refused (error 111)
I20260927 15:16:16.267118 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.266733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39861 (local address 127.0.0.1:54676)
0927 15:16:16.266874 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.266947 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 15:16:16.267295 23885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.125:39861: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39861: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.125.keytab.
Entry for principal kudu/127.10.33.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.125.keytab.
I20260927 15:16:16.286470 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.125:39861
--webserver_interface=127.10.33.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.125.keytab
--principal=kudu/127.10.33.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:42261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:35267,127.10.33.125:39861,127.10.33.124:41243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:16:16.316679 23885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } attempt: 1
I20260927 15:16:16.317963 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.317449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39861 (local address 127.0.0.1:54682)
0927 15:16:16.317653 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.317801 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 15:16:16.319097 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.318711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39861 (local address 127.0.0.1:54688)
0927 15:16:16.318858 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.318936 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 15:16:16.319301 23885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.125:39861: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39861: connect: Connection refused (error 111)
I20260927 15:16:16.387576 23885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } attempt: 2
I20260927 15:16:16.388833 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.388252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39861 (local address 127.0.0.1:54698)
0927 15:16:16.388457 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.388681 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 15:16:16.389955 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.389547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39861 (local address 127.0.0.1:54704)
0927 15:16:16.389686 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.389769 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 15:16:16.390148 23885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.125:39861: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39861: connect: Connection refused (error 111)
I20260927 15:16:16.481443 23885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } attempt: 3
I20260927 15:16:16.482676 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.482187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39861 (local address 127.0.0.1:54706)
0927 15:16:16.482381 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.482540 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 15:16:16.483762 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.483417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39861 (local address 127.0.0.1:54708)
0927 15:16:16.483552 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.483624 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 15:16:16.484055 23885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.125:39861: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39861: connect: Connection refused (error 111)
W20260927 15:16:16.615090 23892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:16.615388 23892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:16.615477 23892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:16.623173 23892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:16.623298 23892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:16.623344 23892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:16.623400 23892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:16.623443 23892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:16:16.627293 23885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } attempt: 4
I20260927 15:16:16.628470 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.627948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39861 (local address 127.0.0.1:59858)
0927 15:16:16.628246 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.628334 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260927 15:16:16.629532 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.629131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39861 (local address 127.0.0.1:59870)
0927 15:16:16.629296 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.629397 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 15:16:16.629743 23885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.125:39861: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39861: connect: Connection refused (error 111)
I20260927 15:16:16.633919 23892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:42261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:35267,127.10.33.125:39861,127.10.33.124:41243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.125.keytab
--principal=kudu/127.10.33.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.125:39861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.33.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:16.635356 23892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:16.636662 23892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:16.649202 23897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:16.649222 23898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:16.649267 23900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:16.650437 23892 server_base.cc:1034] running on GCE node
I20260927 15:16:16.651496 23892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:16.653309 23892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:16.654637 23892 hybrid_clock.cc:648] HybridClock initialized: now 1790522176654517 us; error 55 us; skew 500 ppm
I20260927 15:16:16.655102 23892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:16 dist-test-slave-f5h6 krb5kdc[23805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522176, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:16.664469 23892 init.cc:377] Logged in from keytab as kudu/127.10.33.125@KRBTEST.COM (short username kudu)
I20260927 15:16:16.666332 23892 webserver.cc:460] Webserver started at http://127.10.33.125:43333/ using document root <none> and password file <none>
I20260927 15:16:16.667239 23892 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:16.667384 23892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:16.667701 23892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:16.671141 23892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-1/data/instance:
uuid: "88498090c2f84e638ee4334955ea47fd"
format_stamp: "Formatted at 2026-09-27 15:16:16 on dist-test-slave-f5h6"
server_key: "7b1b8db6c2aeb4db0accccdb88cec648"
server_key_iv: "6e2619b80cfe974e3cc7abb5103c0fa7"
server_key_version: "encryptionkey@0"
I20260927 15:16:16.672127 23892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-1/wal/instance:
uuid: "88498090c2f84e638ee4334955ea47fd"
format_stamp: "Formatted at 2026-09-27 15:16:16 on dist-test-slave-f5h6"
server_key: "7b1b8db6c2aeb4db0accccdb88cec648"
server_key_iv: "6e2619b80cfe974e3cc7abb5103c0fa7"
server_key_version: "encryptionkey@0"
I20260927 15:16:16.677693 23892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:16:16.681407 23907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:16.682868 23892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:16:16.683053 23892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-1/wal
uuid: "88498090c2f84e638ee4334955ea47fd"
format_stamp: "Formatted at 2026-09-27 15:16:16 on dist-test-slave-f5h6"
server_key: "7b1b8db6c2aeb4db0accccdb88cec648"
server_key_iv: "6e2619b80cfe974e3cc7abb5103c0fa7"
server_key_version: "encryptionkey@0"
I20260927 15:16:16.683244 23892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:16.707463 23892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:16.711910 23892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:16.712201 23892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:16.730345 23892 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.125:39861
I20260927 15:16:16.730386 23958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.125:39861 every 8 connection(s)
I20260927 15:16:16.731729 23892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-1/data/info.pb
I20260927 15:16:16.735508 23959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:16.739169 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 23892
I20260927 15:16:16.739370 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-1/wal/instance
I20260927 15:16:16.739895 10373 external_mini_cluster.cc:1428] Setting key 5131a79ce8849ef120e6e6f1a2e4ec62
I20260927 15:16:16.743714 23959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35267 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:16:16 dist-test-slave-f5h6 krb5kdc[23805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522176, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.125@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.33.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.124@KRBTEST.COM" created.
I20260927 15:16:16.765779 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.747141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46480 (local address 127.10.33.126:35267)
0927 15:16:16.747364 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:16.747370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:16.748700 (+ 1330us) server_negotiation.cc:407] Connection header received
0927 15:16:16.750727 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:16.750734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:16.750819 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:16.750908 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:16.752537 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:16.753205 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:16.754901 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:16.755368 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:16.755481 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:16.759410 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:16.759417 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:16.759421 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:16.759461 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:16.762548 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:16.763406 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:16.763411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:16.763413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:16.763550 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:16.763990 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:16.763994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:16.763996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:16.764289 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:16.764439 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:16.765205 (+ 766us) server_negotiation.cc:299] Negotiation successful
0927 15:16:16.765448 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 15:16:16.766822 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.747675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35267 (local address 127.0.0.1:46480)
0927 15:16:16.748513 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.748560 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:16.750410 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:16.751182 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:16.751191 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:16.751739 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:16.752332 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:16.752349 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:16.753366 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:16.753369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:16.754750 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:16.754763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:16.755604 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:16.755612 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:16.755842 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:16.756611 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:16.756637 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:16.759145 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:16.762724 (+ 3579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:16.762733 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:16.762749 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:16.763257 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:16.763666 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:16.763669 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:16.763672 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:16.763869 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:16.764420 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:16.764429 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:16.764817 (+ 388us) client_negotiation.cc:769] Sending connection context
0927 15:16:16.765073 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 15:16:16.765642 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":357,"threads_started":1}
I20260927 15:16:16.768602 23959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:16:16 dist-test-slave-f5h6 krb5kdc[23805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522176, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.125@KRBTEST.COM for kudu/127.10.33.125@KRBTEST.COM
I20260927 15:16:16.783643 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.769329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39861 (local address 127.0.0.1:59884)
0927 15:16:16.769557 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.769576 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:16.769663 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:16.770747 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:16.770752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:16.771203 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:16.771340 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:16.771348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:16.772671 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:16.772676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:16.773486 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:16.773497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:16.774320 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:16.774325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:16.774511 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:16.775149 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:16.775168 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:16.777582 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:16.780992 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:16.780997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:16.781001 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:16.781380 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:16.781796 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:16.781800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:16.781804 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:16.781990 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:16.782660 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:16.782664 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:16.782804 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 15:16:16.782923 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 15:16:16.783220 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260927 15:16:16.783701 23964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.769598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59884 (local address 127.10.33.125:39861)
0927 15:16:16.770266 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:16.770272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:16.770298 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:16:16.770341 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:16.770346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:16.770588 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:16.770747 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:16.771495 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:16.772545 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:16.773609 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:16.774172 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:16.774280 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:16.777826 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:16.777841 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:16.777889 (+ 48us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:16.777946 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:16.780807 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:16.781506 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:16.781513 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:16.781516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:16.781681 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:16.782103 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:16.782106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:16.782108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:16.782525 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:16.782722 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:16.783138 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 15:16:16.783360 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":385,"threads_started":1}
I20260927 15:16:16.786226 23959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:16:16.787513 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.786997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41243 (local address 127.0.0.1:35364)
0927 15:16:16.787200 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.787329 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 15:16:16.787835 23910 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.124.keytab.
Entry for principal kudu/127.10.33.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.124.keytab.
I20260927 15:16:16.789649 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.789176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41243 (local address 127.0.0.1:35368)
0927 15:16:16.789335 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.789491 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 15:16:16.789848 23959 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41243: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
I20260927 15:16:16.790555 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.124:41243
--webserver_interface=127.10.33.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.124.keytab
--principal=kudu/127.10.33.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:42261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:35267,127.10.33.125:39861,127.10.33.124:41243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:16:16.839250 23959 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } attempt: 1
I20260927 15:16:16.840375 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.839928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41243 (local address 127.0.0.1:35372)
0927 15:16:16.840126 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.840226 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 15:16:16.841560 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.841115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41243 (local address 127.0.0.1:35384)
0927 15:16:16.841316 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.841393 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260927 15:16:16.841780 23959 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41243: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
I20260927 15:16:16.910035 23959 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } attempt: 2
I20260927 15:16:16.911124 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.910676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41243 (local address 127.0.0.1:35396)
0927 15:16:16.910876 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.910970 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 15:16:16.912143 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.911804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41243 (local address 127.0.0.1:35402)
0927 15:16:16.911956 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.912028 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 15:16:16.912361 23959 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41243: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
I20260927 15:16:16.929034 23885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } attempt: 5
Sep 27 15:16:16 dist-test-slave-f5h6 krb5kdc[23805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522176, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for kudu/127.10.33.125@KRBTEST.COM
I20260927 15:16:16.943462 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.929783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39861 (local address 127.0.0.1:59894)
0927 15:16:16.930028 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.930049 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:16.930167 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:16.930634 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:16.930639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:16.931018 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:16.931202 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:16.931211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:16.932608 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:16.932612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:16.933528 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:16.933542 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:16.934645 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:16.934650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:16.934804 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:16.935417 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:16.935435 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:16.937405 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:16.941108 (+ 3703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:16.941118 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:16.941121 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:16.941502 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:16.941951 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:16.941955 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:16.941957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:16.942096 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:16.942744 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:16.942747 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:16.942885 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 15:16:16.942995 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 15:16:16.943215 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 15:16:16.943719 23964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.930006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59894 (local address 127.10.33.125:39861)
0927 15:16:16.930204 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:16.930210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:16.930233 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:16:16.930298 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:16.930304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:16.930398 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:16.930512 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:16.931375 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:16.932460 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:16.933674 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:16.934408 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:16.934550 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:16.937608 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:16.937615 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:16.937620 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:16.937661 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:16.940939 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:16.941661 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:16.941667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:16.941671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:16.941814 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:16.942213 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:16.942217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:16.942220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:16.942623 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:16.942778 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:16.943127 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 15:16:16.943376 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 15:16:16.944710 23885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:16:16.945825 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.945419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41243 (local address 127.0.0.1:35408)
0927 15:16:16.945606 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.945687 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 15:16:16.946753 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.946431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41243 (local address 127.0.0.1:35414)
0927 15:16:16.946581 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.946653 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 15:16:16.946961 23885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41243: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
I20260927 15:16:16.998241 23885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } attempt: 1
I20260927 15:16:16.999492 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:16.998994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41243 (local address 127.0.0.1:35424)
0927 15:16:16.999217 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:16.999322 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 15:16:17.000525 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.000162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41243 (local address 127.0.0.1:35434)
0927 15:16:17.000314 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:17.000397 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 15:16:17.000737 23885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41243: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
I20260927 15:16:17.003572 23959 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } attempt: 3
I20260927 15:16:17.004722 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.004238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41243 (local address 127.0.0.1:35440)
0927 15:16:17.004454 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:17.004570 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 15:16:17.005834 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.005367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41243 (local address 127.0.0.1:35448)
0927 15:16:17.005627 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:17.005703 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260927 15:16:17.006053 23959 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41243: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
I20260927 15:16:17.068955 23885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } attempt: 2
I20260927 15:16:17.070149 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.069615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41243 (local address 127.0.0.1:35456)
0927 15:16:17.069877 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:17.069981 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 15:16:17.071082 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.070737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41243 (local address 127.0.0.1:35460)
0927 15:16:17.070894 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:17.070967 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 15:16:17.071285 23885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41243: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
W20260927 15:16:17.116254 23966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:17.116585 23966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:17.116707 23966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:17.124264 23966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:17.124397 23966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:17.124483 23966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:17.124558 23966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:17.124616 23966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:16:17.134796 23966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:42261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:35267,127.10.33.125:39861,127.10.33.124:41243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.124.keytab
--principal=kudu/127.10.33.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.124:41243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.33.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:17.136263 23966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:17.137626 23966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:16:17.149269 23959 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } attempt: 4
W20260927 15:16:17.149600 23972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:17.149927 23974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:17.149943 23971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:17.150511 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.150011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41243 (local address 127.0.0.1:35468)
0927 15:16:17.150213 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:17.150303 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 15:16:17.150787 23966 server_base.cc:1034] running on GCE node
I20260927 15:16:17.151690 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.151309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41243 (local address 127.0.0.1:35476)
0927 15:16:17.151477 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:17.151581 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 15:16:17.151798 23966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20260927 15:16:17.151913 23959 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41243: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
I20260927 15:16:17.153487 23966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:17.154733 23966 hybrid_clock.cc:648] HybridClock initialized: now 1790522177154680 us; error 38 us; skew 500 ppm
I20260927 15:16:17.155169 23966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:17 dist-test-slave-f5h6 krb5kdc[23805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522177, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:17.158677 23966 init.cc:377] Logged in from keytab as kudu/127.10.33.124@KRBTEST.COM (short username kudu)
I20260927 15:16:17.160145 23966 webserver.cc:460] Webserver started at http://127.10.33.124:34289/ using document root <none> and password file <none>
I20260927 15:16:17.160861 23966 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:17.160974 23966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:17.161260 23966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:17.164377 23966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-2/data/instance:
uuid: "a84f345ea01c4336aa1e59ebe1b1826a"
format_stamp: "Formatted at 2026-09-27 15:16:17 on dist-test-slave-f5h6"
server_key: "ab2a391ac41659e128811cac8690fbab"
server_key_iv: "4a510d71006f745cec5b56365e120bf2"
server_key_version: "encryptionkey@0"
I20260927 15:16:17.165236 23966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-2/wal/instance:
uuid: "a84f345ea01c4336aa1e59ebe1b1826a"
format_stamp: "Formatted at 2026-09-27 15:16:17 on dist-test-slave-f5h6"
server_key: "ab2a391ac41659e128811cac8690fbab"
server_key_iv: "4a510d71006f745cec5b56365e120bf2"
server_key_version: "encryptionkey@0"
I20260927 15:16:17.170902 23966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 15:16:17.174671 23981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:17.176196 23966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:16:17.176388 23966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-2/wal
uuid: "a84f345ea01c4336aa1e59ebe1b1826a"
format_stamp: "Formatted at 2026-09-27 15:16:17 on dist-test-slave-f5h6"
server_key: "ab2a391ac41659e128811cac8690fbab"
server_key_iv: "4a510d71006f745cec5b56365e120bf2"
server_key_version: "encryptionkey@0"
I20260927 15:16:17.176599 23966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:17.177501 23885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } attempt: 3
I20260927 15:16:17.178637 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.178207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41243 (local address 127.0.0.1:35480)
0927 15:16:17.178400 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:17.178482 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 15:16:17.179638 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.179247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41243 (local address 127.0.0.1:35494)
0927 15:16:17.179448 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:17.179518 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260927 15:16:17.179826 23885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41243: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41243: connect: Connection refused (error 111)
I20260927 15:16:17.202118 23966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:17.206483 23966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:17.206756 23966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:17.223824 23966 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.124:41243
I20260927 15:16:17.223838 24032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.124:41243 every 8 connection(s)
I20260927 15:16:17.225140 23966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-2/data/info.pb
I20260927 15:16:17.228783 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:17.232084 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 23966
I20260927 15:16:17.232270 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521968392480-10373-0/minicluster-data/master-2/wal/instance
I20260927 15:16:17.232805 10373 external_mini_cluster.cc:1428] Setting key 81001330ee3c73cb02ab3686acbad181
I20260927 15:16:17.236544 24033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35267 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:16:17 dist-test-slave-f5h6 krb5kdc[23805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.124@KRBTEST.COM
Sep 27 15:16:17 dist-test-slave-f5h6 krb5kdc[23805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522177, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.124@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:16:17.255774 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.240067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46494 (local address 127.10.33.126:35267)
0927 15:16:17.240249 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:17.240255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:17.240874 (+ 619us) server_negotiation.cc:407] Connection header received
0927 15:16:17.241062 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:17.241065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:17.241130 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:17.241217 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:17.242665 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.243551 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.245141 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.245569 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.245728 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:17.249134 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:17.249142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:17.249146 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:17.249185 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:17.252350 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:17.253201 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:17.253205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:17.253207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:17.253373 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:17.253748 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:17.253753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:17.253756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:17.254121 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:17.254296 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:17.255042 (+ 746us) server_negotiation.cc:299] Negotiation successful
0927 15:16:17.255292 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:16:17.256632 24036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.234781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35496 (local address 127.10.33.124:41243)
0927 15:16:17.235527 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:17.235539 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:17.235580 (+ 41us) server_negotiation.cc:407] Connection header received
0927 15:16:17.237211 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:17.237236 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:17.237678 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:17.238101 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:17.239837 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.241578 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.243274 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.244287 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.244430 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:17.248601 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:17.248611 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:17.248624 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:17.248655 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:17.251877 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:17.252766 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:17.252773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:17.252777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:17.252946 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:17.253384 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:17.253389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:17.253392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:17.254166 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:17.254471 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:17.254888 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 15:16:17.255409 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":546,"thread_start_us":339,"threads_started":1}
I20260927 15:16:17.256672 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.240067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35267 (local address 127.0.0.1:46494)
0927 15:16:17.240736 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:17.240773 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:17.240925 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:17.241307 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:17.241314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:17.242127 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:17.242480 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:17.242493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.243715 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.243719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:17.245005 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:17.245019 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.245879 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.245887 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:17.246055 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:17.246700 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:17.246724 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:17.248925 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:17.252508 (+ 3583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:17.252522 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:17.252537 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:17.253058 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:17.253494 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:17.253497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:17.253500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:17.253638 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:17.254290 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:17.254299 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:17.254626 (+ 327us) client_negotiation.cc:769] Sending connection context
0927 15:16:17.254910 (+ 284us) client_negotiation.cc:240] Negotiation successful
0927 15:16:17.255553 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":161,"thread_start_us":264,"threads_started":1}
I20260927 15:16:17.259727 24033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:16:17 dist-test-slave-f5h6 krb5kdc[23805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522177, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.124@KRBTEST.COM for kudu/127.10.33.125@KRBTEST.COM
Sep 27 15:16:17 dist-test-slave-f5h6 krb5kdc[23805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:16:17.274230 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.260382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39861 (local address 127.0.0.1:59900)
0927 15:16:17.260580 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:17.260598 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:17.260697 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:17.260996 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:17.261000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:17.261422 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:17.261639 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:17.261651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.263041 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.263045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:17.264314 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:17.264329 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.265136 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.265140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:17.265283 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:17.265822 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:17.265840 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:17.268128 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:17.271596 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:17.271600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:17.271605 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:17.272131 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:17.272543 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:17.272548 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:17.272552 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:17.272738 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:17.273252 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:17.273256 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:17.273402 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 15:16:17.273525 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 15:16:17.273830 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 15:16:17.274282 23964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.260519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59900 (local address 127.10.33.125:39861)
0927 15:16:17.260716 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:17.260721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:17.260738 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:16:17.260791 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:17.260795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:17.260852 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:17.260941 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:17.261807 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.262897 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.264467 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.264975 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.265107 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:17.268345 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:17.268352 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:17.268357 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:17.268410 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:17.271453 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:17.272273 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:17.272277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:17.272280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:17.272424 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:17.272838 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:17.272842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:17.272844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:17.273118 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:17.273267 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:17.273689 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 15:16:17.273944 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 15:16:17.275549 24033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:16:17.275878 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.260040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46502 (local address 127.10.33.126:35267)
0927 15:16:17.260204 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:17.260210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:17.260229 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:16:17.260288 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:17.260293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:17.260385 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:17.260491 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:17.262200 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.263209 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.264814 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.265424 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.265538 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:17.270270 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:17.270275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:17.270278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:17.270311 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:17.273014 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:17.273751 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:17.273757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:17.273761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:17.273966 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:17.274398 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:17.274404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:17.274408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:17.274714 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:17.274929 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:17.275311 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 15:16:17.275547 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 15:16:17 dist-test-slave-f5h6 krb5kdc[23805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522177, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.124@KRBTEST.COM for kudu/127.10.33.124@KRBTEST.COM
Sep 27 15:16:17 dist-test-slave-f5h6 krb5kdc[23805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.125@KRBTEST.COM
I20260927 15:16:17.290060 24036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.276668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35502 (local address 127.10.33.124:41243)
0927 15:16:17.276843 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:17.276849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:17.277071 (+ 222us) server_negotiation.cc:407] Connection header received
0927 15:16:17.277204 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:17.277210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:17.277270 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:17.277557 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:17.278205 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.278882 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.280404 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.280857 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.280962 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:17.284316 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:17.284320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:17.284323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:17.284350 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:17.287029 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:17.287818 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:17.287822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:17.287824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:17.287945 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:17.288352 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:17.288358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:17.288362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:17.288734 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:17.288852 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:17.289323 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 15:16:17.289757 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":230,"server-negotiator.queue_time_us":68}
I20260927 15:16:17.290066 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.276729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41243 (local address 127.0.0.1:35502)
0927 15:16:17.276981 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:17.276999 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:17.277106 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:17.277390 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:17.277394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:17.277900 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:17.278100 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:17.278112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.278997 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.279003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:17.280271 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:17.280285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.280976 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.280982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:17.281169 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:17.281783 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:17.281805 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:17.284152 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:17.287162 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:17.287167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:17.287170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:17.287714 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:17.288047 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:17.288052 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:17.288056 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:17.288211 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:17.288883 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:17.288888 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:17.289061 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 15:16:17.289233 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 15:16:17.289622 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260927 15:16:17.292918 23964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.277464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59916 (local address 127.10.33.125:39861)
0927 15:16:17.277782 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:17.277788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:17.277803 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:16:17.277850 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:17.277877 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:17.277944 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:17.278042 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:17.279198 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.280222 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.281567 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.282193 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.282314 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:17.286708 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:17.286714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:17.286717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:17.286750 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:17.290105 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:17.290913 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:17.290918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:17.290922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:17.291118 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:17.291496 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:17.291500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:17.291502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:17.291767 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:17.291990 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:17.292434 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 15:16:17.292613 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20260927 15:16:17.295651 24033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a: Bootstrap starting.
I20260927 15:16:17.297614 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:16:17.299453 24033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:17.300791 24033 log.cc:824] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:17.304131 24033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a: No bootstrap required, opened a new log
I20260927 15:16:17.314886 24033 raft_consensus.cc:348] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb58db88c164300ac2e07b35594cad8" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35267 } } peers { permanent_uuid: "88498090c2f84e638ee4334955ea47fd" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } } peers { permanent_uuid: "a84f345ea01c4336aa1e59ebe1b1826a" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } }
I20260927 15:16:17.315352 24033 raft_consensus.cc:374] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:17.315477 24033 raft_consensus.cc:729] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a84f345ea01c4336aa1e59ebe1b1826a, State: Initialized, Role: FOLLOWER
I20260927 15:16:17.316299 24033 consensus_queue.cc:260] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb58db88c164300ac2e07b35594cad8" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35267 } } peers { permanent_uuid: "88498090c2f84e638ee4334955ea47fd" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } } peers { permanent_uuid: "a84f345ea01c4336aa1e59ebe1b1826a" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } }
I20260927 15:16:17.318500 24046 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb58db88c164300ac2e07b35594cad8" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35267 } } peers { permanent_uuid: "88498090c2f84e638ee4334955ea47fd" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } } peers { permanent_uuid: "a84f345ea01c4336aa1e59ebe1b1826a" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } } }
I20260927 15:16:17.318935 24046 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:16:17.320104 24033 sys_catalog.cc:564] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:17.324038 23964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.305035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.10.33.125:39861)
0927 15:16:17.305314 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:17.305320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:17.305808 (+ 488us) server_negotiation.cc:407] Connection header received
0927 15:16:17.306088 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:17.306094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:17.306170 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:17.306240 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:17.308550 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.309567 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.311762 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.312376 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.312471 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:17.316604 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:17.316611 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:17.316616 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:17.316658 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:17.320627 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:17.321577 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:17.321583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:17.321587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:17.321755 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:17.322187 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:17.322192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:17.322194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:17.322587 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:17.322821 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:17.323441 (+ 620us) server_negotiation.cc:299] Negotiation successful
0927 15:16:17.323694 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260927 15:16:17.324361 24036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.303511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35504 (local address 127.10.33.124:41243)
0927 15:16:17.303663 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:17.303669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:17.305068 (+ 1399us) server_negotiation.cc:407] Connection header received
0927 15:16:17.305153 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:17.305158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:17.305237 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:17.305320 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:17.308072 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.309239 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.311639 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.312373 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.312470 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:17.316599 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:17.316607 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:17.316611 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:17.316658 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:17.320962 (+ 4304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:17.321817 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:17.321822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:17.321826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:17.322024 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:17.322426 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:17.322430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:17.322434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:17.322806 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:17.323022 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:17.323825 (+ 803us) server_negotiation.cc:299] Negotiation successful
0927 15:16:17.324044 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 15:16:17.326607 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.303341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46514 (local address 127.10.33.126:35267)
0927 15:16:17.303699 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:17.303706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:17.304298 (+ 592us) server_negotiation.cc:407] Connection header received
0927 15:16:17.305215 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:17.305221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:17.305305 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:17.305414 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:17.308051 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.309134 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.311424 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.312094 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.312196 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:17.316974 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:17.316981 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:17.316986 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:17.317035 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:17.322854 (+ 5819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:17.324031 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:17.324038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:17.324042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:17.324229 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:17.324668 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:17.324674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:17.324677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:17.325063 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:17.325311 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:17.325954 (+ 643us) server_negotiation.cc:299] Negotiation successful
0927 15:16:17.326224 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238}
W20260927 15:16:17.335279 24057 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:16:17.335430 24057 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:16:17.333117 23966 master_runner.cc:186] Error getting master registration for 127.10.33.126:35267: OK, instance_id { permanent_uuid: "7cb58db88c164300ac2e07b35594cad8" instance_seqno: 1790522176216127 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:16:17.341547 23966 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:16:17.357019 23885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } attempt: 4
Sep 27 15:16:17 dist-test-slave-f5h6 krb5kdc[23805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522176, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for kudu/127.10.33.124@KRBTEST.COM
I20260927 15:16:17.371323 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.357664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41243 (local address 127.0.0.1:35506)
0927 15:16:17.357875 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:17.357899 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:17.358012 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:17.358364 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:17.358368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:17.358795 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:17.358958 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:17.358967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.360409 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.360413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:17.361422 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:17.361436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.362436 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.362441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:17.362668 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:17.363315 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:17.363333 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:17.365800 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:17.368963 (+ 3163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:17.368968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:17.368973 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:17.369334 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:17.369695 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:17.369700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:17.369704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:17.369850 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:17.370374 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:17.370378 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:17.370546 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 15:16:17.370657 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 15:16:17.370942 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 15:16:17.371351 24036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.357806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35506 (local address 127.10.33.124:41243)
0927 15:16:17.358037 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:17.358042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:17.358062 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:16:17.358115 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:17.358120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:17.358208 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:17.358319 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:17.359094 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.360240 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.361576 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.362283 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.362419 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:17.366076 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:17.366083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:17.366087 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:17.366124 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:17.368841 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:17.369445 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:17.369449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:17.369452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:17.369593 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:17.369966 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:17.369979 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:17.369983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:17.370264 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:17.370396 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:17.370791 (+ 395us) server_negotiation.cc:299] Negotiation successful
0927 15:16:17.371010 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260927 15:16:17.377913 23885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8: Bootstrap starting.
I20260927 15:16:17.382169 23885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:17.383486 23885 log.cc:824] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:17.386888 23885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8: No bootstrap required, opened a new log
I20260927 15:16:17.395341 23885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb58db88c164300ac2e07b35594cad8" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35267 } } peers { permanent_uuid: "88498090c2f84e638ee4334955ea47fd" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } } peers { permanent_uuid: "a84f345ea01c4336aa1e59ebe1b1826a" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } }
I20260927 15:16:17.395687 23885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:17.395812 23885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cb58db88c164300ac2e07b35594cad8, State: Initialized, Role: FOLLOWER
I20260927 15:16:17.396355 23885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb58db88c164300ac2e07b35594cad8" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35267 } } peers { permanent_uuid: "88498090c2f84e638ee4334955ea47fd" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } } peers { permanent_uuid: "a84f345ea01c4336aa1e59ebe1b1826a" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } }
I20260927 15:16:17.397254 24060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb58db88c164300ac2e07b35594cad8" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35267 } } peers { permanent_uuid: "88498090c2f84e638ee4334955ea47fd" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } } peers { permanent_uuid: "a84f345ea01c4336aa1e59ebe1b1826a" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } } }
I20260927 15:16:17.397662 24060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:16:17.398300 23885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 15:16:17.409511 24071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:16:17.409729 24071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:16:17.408347 23818 master_runner.cc:186] Error getting master registration for 127.10.33.125:39861: OK, instance_id { permanent_uuid: "88498090c2f84e638ee4334955ea47fd" instance_seqno: 1790522176725110 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:16:17.415912 23818 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:16:17.431207 24060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:16:17.431567 24060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb58db88c164300ac2e07b35594cad8" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35267 } } peers { permanent_uuid: "88498090c2f84e638ee4334955ea47fd" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } } peers { permanent_uuid: "a84f345ea01c4336aa1e59ebe1b1826a" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } }
I20260927 15:16:17.433102 24060 leader_election.cc:288] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88498090c2f84e638ee4334955ea47fd (127.10.33.125:39861), a84f345ea01c4336aa1e59ebe1b1826a (127.10.33.124:41243)
I20260927 15:16:17.434119 24008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7cb58db88c164300ac2e07b35594cad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a84f345ea01c4336aa1e59ebe1b1826a" is_pre_election: true
I20260927 15:16:17.434767 24008 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cb58db88c164300ac2e07b35594cad8 in term 0.
I20260927 15:16:17.435377 23836 leader_election.cc:302] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cb58db88c164300ac2e07b35594cad8, a84f345ea01c4336aa1e59ebe1b1826a; no voters:
I20260927 15:16:17.435765 24060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:16:17.435911 24060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:16:17.436015 24060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:17.439410 24060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb58db88c164300ac2e07b35594cad8" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35267 } } peers { permanent_uuid: "88498090c2f84e638ee4334955ea47fd" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } } peers { permanent_uuid: "a84f345ea01c4336aa1e59ebe1b1826a" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } }
I20260927 15:16:17.440179 24060 leader_election.cc:288] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [CANDIDATE]: Term 1 election: Requested vote from peers 88498090c2f84e638ee4334955ea47fd (127.10.33.125:39861), a84f345ea01c4336aa1e59ebe1b1826a (127.10.33.124:41243)
I20260927 15:16:17.440909 24008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7cb58db88c164300ac2e07b35594cad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a84f345ea01c4336aa1e59ebe1b1826a"
I20260927 15:16:17.441164 24008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:17.444650 24008 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cb58db88c164300ac2e07b35594cad8 in term 1.
I20260927 15:16:17.445173 23836 leader_election.cc:302] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cb58db88c164300ac2e07b35594cad8, a84f345ea01c4336aa1e59ebe1b1826a; no voters:
I20260927 15:16:17.445578 24060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:17.447944 24060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [term 1 LEADER]: Becoming Leader. State: Replica: 7cb58db88c164300ac2e07b35594cad8, State: Running, Role: LEADER
I20260927 15:16:17.448606 24060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb58db88c164300ac2e07b35594cad8" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35267 } } peers { permanent_uuid: "88498090c2f84e638ee4334955ea47fd" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } } peers { permanent_uuid: "a84f345ea01c4336aa1e59ebe1b1826a" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } }
I20260927 15:16:17.451120 23959 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } attempt: 5
I20260927 15:16:17.452282 24073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cb58db88c164300ac2e07b35594cad8. Latest consensus state: current_term: 1 leader_uuid: "7cb58db88c164300ac2e07b35594cad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb58db88c164300ac2e07b35594cad8" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35267 } } peers { permanent_uuid: "88498090c2f84e638ee4334955ea47fd" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } } peers { permanent_uuid: "a84f345ea01c4336aa1e59ebe1b1826a" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } } }
I20260927 15:16:17.452709 24073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [sys.catalog]: This master's current role is: LEADER
W20260927 15:16:17.455798 23839 tablet.cc:2369] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:16:17.458907 23987 tablet.cc:2369] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 15:16:17 dist-test-slave-f5h6 krb5kdc[23805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522176, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.125@KRBTEST.COM for kudu/127.10.33.124@KRBTEST.COM
I20260927 15:16:17.469640 24075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:16:17.472151 24075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:16:17.477031 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.451795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41243 (local address 127.0.0.1:35522)
0927 15:16:17.451996 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:17.452020 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:17.452150 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:17.454754 (+ 2604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:17.454760 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:17.455315 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:17.455534 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:17.455547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.457129 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.457135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:17.458760 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:17.458782 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.459724 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.459730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:17.459980 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:17.460712 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:17.460738 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:17.463561 (+ 2823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:17.473298 (+ 9737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:17.473314 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:17.473319 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:17.473904 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:17.474665 (+ 761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:17.474670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:17.474673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:17.474848 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:17.475987 (+ 1139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:17.475991 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:17.476147 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 15:16:17.476279 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 15:16:17.476621 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 15:16:17.479825 24036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:17.454057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35522 (local address 127.10.33.124:41243)
0927 15:16:17.454245 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:17.454251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:17.454269 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:16:17.454333 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:17.454339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:17.454421 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:17.454526 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:17.455864 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.456967 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.458917 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:17.459577 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:17.459694 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:17.463822 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:17.463830 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:17.463836 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:17.463886 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:17.473055 (+ 9169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:17.474156 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:17.474161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:17.474165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:17.474445 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:17.475046 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:17.475086 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:17.475089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:17.475788 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:17.477888 (+ 2100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:17.478349 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 15:16:17.479005 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:16:17.486681 23959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88498090c2f84e638ee4334955ea47fd: Bootstrap starting.
I20260927 15:16:17.488212 24008 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a [term 1 FOLLOWER]: Refusing update from remote peer 7cb58db88c164300ac2e07b35594cad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:17.489143 24073 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a84f345ea01c4336aa1e59ebe1b1826a" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:17.495769 23959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88498090c2f84e638ee4334955ea47fd: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:17.498374 23959 log.cc:824] T 00000000000000000000000000000000 P 88498090c2f84e638ee4334955ea47fd: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:17.505188 24046 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cb58db88c164300ac2e07b35594cad8. Latest consensus state: current_term: 1 leader_uuid: "7cb58db88c164300ac2e07b35594cad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb58db88c164300ac2e07b35594cad8" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35267 } } peers { permanent_uuid: "88498090c2f84e638ee4334955ea47fd" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } } peers { permanent_uuid: "a84f345ea01c4336aa1e59ebe1b1826a" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } } }
I20260927 15:16:17.505525 24046 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:16:17.509294 24076 mvcc.cc:205] Tried to move back new op lower bound from 7333978838978322432 to 7333978838834339840. Current Snapshot: MvccSnapshot[applied={T|T < 7333978838978322432}]
I20260927 15:16:17.511300 24077 mvcc.cc:205] Tried to move back new op lower bound from 7333978838978322432 to 7333978838834339840. Current Snapshot: MvccSnapshot[applied={T|T < 7333978838978322432}]
I20260927 15:16:17.514343 24046 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7cb58db88c164300ac2e07b35594cad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb58db88c164300ac2e07b35594cad8" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35267 } } peers { permanent_uuid: "88498090c2f84e638ee4334955ea47fd" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } } peers { permanent_uuid: "a84f345ea01c4336aa1e59ebe1b1826a" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } } }
I20260927 15:16:17.514576 24046 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:16:17.517227 23959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88498090c2f84e638ee4334955ea47fd: No bootstrap required, opened a new log
I20260927 15:16:17.530404 24060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7cb58db88c164300ac2e07b35594cad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb58db88c164300ac2e07b35594cad8" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35267 } } peers { permanent_uuid: "88498090c2f84e638ee4334955ea47fd" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } } peers { permanent_uuid: "a84f345ea01c4336aa1e59ebe1b1826a" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } } }
I20260927 15:16:17.530756 24060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:17.532689 24075 catalog_manager.cc:1347] Generated new cluster ID: 78efbc7d369e4f6fb6e0bff0e07f03f9
I20260927 15:16:17.532788 24075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:16:17.537842 23959 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88498090c2f84e638ee4334955ea47fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb58db88c164300ac2e07b35594cad8" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35267 } } peers { permanent_uuid: "88498090c2f84e638ee4334955ea47fd" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } } peers { permanent_uuid: "a84f345ea01c4336aa1e59ebe1b1826a" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } }
I20260927 15:16:17.538461 23959 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88498090c2f84e638ee4334955ea47fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:17.538620 23959 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88498090c2f84e638ee4334955ea47fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88498090c2f84e638ee4334955ea47fd, State: Initialized, Role: FOLLOWER
I20260927 15:16:17.539466 23959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88498090c2f84e638ee4334955ea47fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb58db88c164300ac2e07b35594cad8" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35267 } } peers { permanent_uuid: "88498090c2f84e638ee4334955ea47fd" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } } peers { permanent_uuid: "a84f345ea01c4336aa1e59ebe1b1826a" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } }
I20260927 15:16:17.550455 24082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88498090c2f84e638ee4334955ea47fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb58db88c164300ac2e07b35594cad8" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35267 } } peers { permanent_uuid: "88498090c2f84e638ee4334955ea47fd" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } } peers { permanent_uuid: "a84f345ea01c4336aa1e59ebe1b1826a" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } } }
I20260927 15:16:17.550876 24082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88498090c2f84e638ee4334955ea47fd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:16:17.556124 23959 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88498090c2f84e638ee4334955ea47fd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:17.556499 23932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88498090c2f84e638ee4334955ea47fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:17.560468 23932 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88498090c2f84e638ee4334955ea47fd [term 1 FOLLOWER]: Refusing update from remote peer 7cb58db88c164300ac2e07b35594cad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:17.566532 24075 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:16:17.568929 24075 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:16:17.560678 23934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7cb58db88c164300ac2e07b35594cad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88498090c2f84e638ee4334955ea47fd" is_pre_election: true
I20260927 15:16:17.561046 23933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7cb58db88c164300ac2e07b35594cad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88498090c2f84e638ee4334955ea47fd"
I20260927 15:16:17.571190 24060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88498090c2f84e638ee4334955ea47fd" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:17.583747 24075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8: Generated new TSK 0
I20260927 15:16:17.584882 24075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:16:17.673795 24082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88498090c2f84e638ee4334955ea47fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cb58db88c164300ac2e07b35594cad8. Latest consensus state: current_term: 1 leader_uuid: "7cb58db88c164300ac2e07b35594cad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb58db88c164300ac2e07b35594cad8" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35267 } } peers { permanent_uuid: "88498090c2f84e638ee4334955ea47fd" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } } peers { permanent_uuid: "a84f345ea01c4336aa1e59ebe1b1826a" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } } }
I20260927 15:16:17.674206 24082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88498090c2f84e638ee4334955ea47fd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:16:17.674162 24073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7cb58db88c164300ac2e07b35594cad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb58db88c164300ac2e07b35594cad8" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35267 } } peers { permanent_uuid: "88498090c2f84e638ee4334955ea47fd" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } } peers { permanent_uuid: "a84f345ea01c4336aa1e59ebe1b1826a" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } } }
I20260927 15:16:17.674467 24073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cb58db88c164300ac2e07b35594cad8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:17.674795 24088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88498090c2f84e638ee4334955ea47fd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7cb58db88c164300ac2e07b35594cad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb58db88c164300ac2e07b35594cad8" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35267 } } peers { permanent_uuid: "88498090c2f84e638ee4334955ea47fd" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39861 } } peers { permanent_uuid: "a84f345ea01c4336aa1e59ebe1b1826a" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41243 } } }
I20260927 15:16:17.675271 24088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88498090c2f84e638ee4334955ea47fd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:16:17.722572 24103 catalog_manager.cc:1259] Loaded cluster ID: 78efbc7d369e4f6fb6e0bff0e07f03f9
I20260927 15:16:17.722683 24103 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 88498090c2f84e638ee4334955ea47fd: loading cluster ID for follower catalog manager: success
I20260927 15:16:17.742255 24103 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 88498090c2f84e638ee4334955ea47fd: acquiring CA information for follower catalog manager: success
I20260927 15:16:17.745128 24103 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 88498090c2f84e638ee4334955ea47fd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:16:18.337461 24057 catalog_manager.cc:1259] Loaded cluster ID: 78efbc7d369e4f6fb6e0bff0e07f03f9
I20260927 15:16:18.337571 24057 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a: loading cluster ID for follower catalog manager: success
I20260927 15:16:18.339931 24057 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a: acquiring CA information for follower catalog manager: success
I20260927 15:16:18.341660 24057 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a84f345ea01c4336aa1e59ebe1b1826a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:16:18.749322 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 23818
I20260927 15:16:18.750481 23880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:18.858774 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 23818
I20260927 15:16:18.876647 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 23892
I20260927 15:16:18.877717 23954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:18.983623 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 23892
I20260927 15:16:19.000579 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 23966
I20260927 15:16:19.001760 24028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:19.109148 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 23966
2026-09-27T15:16:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4614 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:16:19 dist-test-slave-f5h6 krb5kdc[24110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:16:19 dist-test-slave-f5h6 krb5kdc[24110](info): set up 2 sockets
Sep 27 15:16:19 dist-test-slave-f5h6 krb5kdc[24110](info): commencing operation
krb5kdc: starting...
W20260927 15:16:20.138177 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:16:20 dist-test-slave-f5h6 krb5kdc[24110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:16:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:16:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:16:20.361840 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:39627
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:45415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:39627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:20.697134 24123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:20.697475 24123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:20.697616 24123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:20.705103 24123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:20.705245 24123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:20.705315 24123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:20.705404 24123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:20.705467 24123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:16:20.715680 24123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:45415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:39627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:39627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:20.717274 24123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:20.718640 24123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:20.730908 24128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:20.730886 24129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:20.730953 24131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:20.731570 24123 server_base.cc:1034] running on GCE node
I20260927 15:16:20.732391 24123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:20.733951 24123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:20.735212 24123 hybrid_clock.cc:648] HybridClock initialized: now 1790522180735148 us; error 48 us; skew 500 ppm
I20260927 15:16:20.735602 24123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:20 dist-test-slave-f5h6 krb5kdc[24110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522180, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:20.738997 24123 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:16:20.740901 24123 webserver.cc:460] Webserver started at http://127.10.33.126:35311/ using document root <none> and password file <none>
I20260927 15:16:20.741694 24123 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:20.741808 24123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:20.742177 24123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:20.745283 24123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "19ff72d2758d4a1c9ac0ad675798ab6e"
format_stamp: "Formatted at 2026-09-27 15:16:20 on dist-test-slave-f5h6"
server_key: "c417c33bdf0e1882cf50e3711ad61cc7"
server_key_iv: "174d6d726b309bfaa584708e75fededc"
server_key_version: "encryptionkey@0"
I20260927 15:16:20.746127 24123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "19ff72d2758d4a1c9ac0ad675798ab6e"
format_stamp: "Formatted at 2026-09-27 15:16:20 on dist-test-slave-f5h6"
server_key: "c417c33bdf0e1882cf50e3711ad61cc7"
server_key_iv: "174d6d726b309bfaa584708e75fededc"
server_key_version: "encryptionkey@0"
I20260927 15:16:20.751471 24123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 15:16:20.755132 24138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:20.756603 24123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:16:20.756771 24123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "19ff72d2758d4a1c9ac0ad675798ab6e"
format_stamp: "Formatted at 2026-09-27 15:16:20 on dist-test-slave-f5h6"
server_key: "c417c33bdf0e1882cf50e3711ad61cc7"
server_key_iv: "174d6d726b309bfaa584708e75fededc"
server_key_version: "encryptionkey@0"
I20260927 15:16:20.756987 24123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:20.775430 24123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:20.780085 24123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:20.780391 24123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:20.798695 24123 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:39627
I20260927 15:16:20.798714 24189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:39627 every 8 connection(s)
I20260927 15:16:20.800100 24123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:16:20.804073 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 24123
I20260927 15:16:20.804123 24190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:20.804308 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:16:20.805143 10373 external_mini_cluster.cc:1428] Setting key ee3de911f52432a8e57ac95b30fc36ed
I20260927 15:16:20.819317 24190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e: Bootstrap starting.
Sep 27 15:16:20 dist-test-slave-f5h6 krb5kdc[24110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:16:20.822875 24190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:20.824052 24190 log.cc:824] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:20.826939 24190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e: No bootstrap required, opened a new log
I20260927 15:16:20.829099 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:20.807054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57358 (local address 127.10.33.126:39627)
0927 15:16:20.807896 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:20.807908 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:20.807950 (+ 42us) server_negotiation.cc:407] Connection header received
0927 15:16:20.809543 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:20.809569 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:20.810067 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:20.810429 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:20.812280 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:20.813593 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:20.815506 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:20.816301 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:20.816459 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:20.821394 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:20.821403 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:20.821418 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:20.821469 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:20.824822 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:20.825613 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:20.825621 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:20.825627 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:20.825795 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:20.826210 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:20.826214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:20.826217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:20.826754 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:20.827045 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:20.827567 (+ 522us) server_negotiation.cc:299] Negotiation successful
0927 15:16:20.827985 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":393,"threads_started":1}
I20260927 15:16:20.833375 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:16:20.835914 24190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ff72d2758d4a1c9ac0ad675798ab6e" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39627 } }
I20260927 15:16:20.836510 24190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:20.836639 24190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19ff72d2758d4a1c9ac0ad675798ab6e, State: Initialized, Role: FOLLOWER
I20260927 15:16:20.837224 24190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ff72d2758d4a1c9ac0ad675798ab6e" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39627 } }
I20260927 15:16:20.837450 24190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:16:20.837577 24190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:16:20.837723 24190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:20.839978 24190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ff72d2758d4a1c9ac0ad675798ab6e" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39627 } }
I20260927 15:16:20.840500 24190 leader_election.cc:302] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19ff72d2758d4a1c9ac0ad675798ab6e; no voters:
I20260927 15:16:20.841145 24190 leader_election.cc:288] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:16:20.841297 24195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:20.842070 24195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e [term 1 LEADER]: Becoming Leader. State: Replica: 19ff72d2758d4a1c9ac0ad675798ab6e, State: Running, Role: LEADER
I20260927 15:16:20.842643 24195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ff72d2758d4a1c9ac0ad675798ab6e" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39627 } }
I20260927 15:16:20.843323 24190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:20.845504 24197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19ff72d2758d4a1c9ac0ad675798ab6e. Latest consensus state: current_term: 1 leader_uuid: "19ff72d2758d4a1c9ac0ad675798ab6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ff72d2758d4a1c9ac0ad675798ab6e" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39627 } } }
I20260927 15:16:20.845697 24196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19ff72d2758d4a1c9ac0ad675798ab6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ff72d2758d4a1c9ac0ad675798ab6e" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39627 } } }
I20260927 15:16:20.846050 24197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:20.846138 24196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:20.846973 24201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:16:20.852541 24201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:16:20.862361 24201 catalog_manager.cc:1347] Generated new cluster ID: cf661a4cb4b04e6a864c94038bfeb688
I20260927 15:16:20.862493 24201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:16:20.870772 24201 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:16:20.872128 24201 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:16:20.880657 24201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19ff72d2758d4a1c9ac0ad675798ab6e: Generated new TSK 0
I20260927 15:16:20.881544 24201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:16:20.894363 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 24123
I20260927 15:16:20.895627 24185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:20.984961 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 24123
2026-09-27T15:16:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1872 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:16:21 dist-test-slave-f5h6 krb5kdc[24218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:16:21 dist-test-slave-f5h6 krb5kdc[24218](info): set up 2 sockets
Sep 27 15:16:21 dist-test-slave-f5h6 krb5kdc[24218](info): commencing operation
krb5kdc: starting...
W20260927 15:16:21.994673 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:16:22 dist-test-slave-f5h6 krb5kdc[24218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:16:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:16:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:16:22.223701 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:35165
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:35165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:22.551003 24231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:22.551311 24231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:22.551412 24231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:22.558743 24231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:22.558856 24231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:22.558908 24231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:22.558965 24231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:22.559008 24231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:16:22.569141 24231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:35165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:35165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:22.570482 24231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:22.571738 24231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:22.583243 24236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:22.583216 24237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:22.583171 24239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:22.584408 24231 server_base.cc:1034] running on GCE node
I20260927 15:16:22.585192 24231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:22.586725 24231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:22.587960 24231 hybrid_clock.cc:648] HybridClock initialized: now 1790522182587893 us; error 49 us; skew 500 ppm
I20260927 15:16:22.588280 24231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:22 dist-test-slave-f5h6 krb5kdc[24218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522182, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:22.596724 24231 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:16:22.598315 24231 webserver.cc:460] Webserver started at http://127.10.33.126:41623/ using document root <none> and password file <none>
I20260927 15:16:22.598977 24231 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:22.599079 24231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:22.599342 24231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:22.602396 24231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "e5a5cd3ea7144cac9eb3e472f7f51295"
format_stamp: "Formatted at 2026-09-27 15:16:22 on dist-test-slave-f5h6"
server_key: "2e5ed7f594e28d5dcc658b6c680dac79"
server_key_iv: "e8c8e4fa99ecbd10fd1052170a529cb3"
server_key_version: "encryptionkey@0"
I20260927 15:16:22.603116 24231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "e5a5cd3ea7144cac9eb3e472f7f51295"
format_stamp: "Formatted at 2026-09-27 15:16:22 on dist-test-slave-f5h6"
server_key: "2e5ed7f594e28d5dcc658b6c680dac79"
server_key_iv: "e8c8e4fa99ecbd10fd1052170a529cb3"
server_key_version: "encryptionkey@0"
I20260927 15:16:22.608484 24231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:16:22.612076 24246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:22.613446 24231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 15:16:22.613601 24231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "e5a5cd3ea7144cac9eb3e472f7f51295"
format_stamp: "Formatted at 2026-09-27 15:16:22 on dist-test-slave-f5h6"
server_key: "2e5ed7f594e28d5dcc658b6c680dac79"
server_key_iv: "e8c8e4fa99ecbd10fd1052170a529cb3"
server_key_version: "encryptionkey@0"
I20260927 15:16:22.613782 24231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:22.629169 24231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:22.633312 24231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:22.633546 24231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:22.652275 24231 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:35165
I20260927 15:16:22.652299 24297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:35165 every 8 connection(s)
I20260927 15:16:22.653719 24231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:16:22.655930 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 24231
I20260927 15:16:22.656243 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:16:22.656924 10373 external_mini_cluster.cc:1428] Setting key 0474fddfbec8a777e64fa14642278653
I20260927 15:16:22.658295 24298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 15:16:22 dist-test-slave-f5h6 krb5kdc[24218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:16:22.674603 24298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295: Bootstrap starting.
I20260927 15:16:22.678967 24298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:22.680120 24298 log.cc:824] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:22.683290 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:22.659220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44244 (local address 127.10.33.126:35165)
0927 15:16:22.660112 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:22.660126 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:22.660169 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:16:22.661794 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:22.661819 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:22.662315 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:22.662694 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:22.664774 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:22.666327 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:22.668325 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:22.669092 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:22.669251 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:22.675124 (+ 5873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:22.675138 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:22.675160 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:22.675245 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:22.679013 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:22.679832 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:22.679840 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:22.679846 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:22.680021 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:22.680448 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:22.680451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:22.680453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:22.681030 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:22.681355 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:22.681889 (+ 534us) server_negotiation.cc:299] Negotiation successful
0927 15:16:22.682305 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":396,"threads_started":1}
I20260927 15:16:22.683385 24298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295: No bootstrap required, opened a new log
I20260927 15:16:22.692211 24298 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35165 } }
I20260927 15:16:22.692708 24298 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:22.692814 24298 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5a5cd3ea7144cac9eb3e472f7f51295, State: Initialized, Role: FOLLOWER
I20260927 15:16:22.693478 24298 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35165 } }
I20260927 15:16:22.693739 24298 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:16:22.693915 24298 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:16:22.694141 24298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:22.696960 24298 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35165 } }
I20260927 15:16:22.697508 24298 leader_election.cc:302] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5a5cd3ea7144cac9eb3e472f7f51295; no voters:
I20260927 15:16:22.698165 24298 leader_election.cc:288] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:16:22.698292 24303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:22.699028 24303 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 1 LEADER]: Becoming Leader. State: Replica: e5a5cd3ea7144cac9eb3e472f7f51295, State: Running, Role: LEADER
I20260927 15:16:22.699551 24303 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35165 } }
I20260927 15:16:22.700356 24298 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:22.702316 24304 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35165 } } }
I20260927 15:16:22.702762 24304 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:22.702628 24305 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5a5cd3ea7144cac9eb3e472f7f51295. Latest consensus state: current_term: 1 leader_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35165 } } }
I20260927 15:16:22.702948 24305 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:22.704331 24311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:16:22.709766 24311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:16:22.720149 24311 catalog_manager.cc:1347] Generated new cluster ID: ed96f4cbbb8840d0b2166907e5a90c55
I20260927 15:16:22.720283 24311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:16:22.745688 24311 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:16:22.747305 24311 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:16:22.759858 24311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295: Generated new TSK 0
I20260927 15:16:22.760754 24311 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.33.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
I20260927 15:16:22.826362 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:35165
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:23.168957 24324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:23.169257 24324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:23.169377 24324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:23.176795 24324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:23.176920 24324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:23.177078 24324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:16:23.187183 24324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:35165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:23.188516 24324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:23.189873 24324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:23.203157 24330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:23.203356 24332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:23.203105 24329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:23.208734 24324 server_base.cc:1034] running on GCE node
I20260927 15:16:23.209430 24324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:23.210510 24324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:23.211730 24324 hybrid_clock.cc:648] HybridClock initialized: now 1790522183211682 us; error 49 us; skew 500 ppm
I20260927 15:16:23.212055 24324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:23 dist-test-slave-f5h6 krb5kdc[24218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522183, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:23.215389 24324 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:16:23.217087 24324 webserver.cc:460] Webserver started at http://127.10.33.65:39529/ using document root <none> and password file <none>
I20260927 15:16:23.217818 24324 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:23.217970 24324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:23.218302 24324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:23.221320 24324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "fa90345f69fa494eb8dbcde852d342e8"
format_stamp: "Formatted at 2026-09-27 15:16:23 on dist-test-slave-f5h6"
server_key: "88fa5f86303d37147141adbad5037929"
server_key_iv: "99e396676dafa35cc9c2d001f343615a"
server_key_version: "encryptionkey@0"
I20260927 15:16:23.222198 24324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "fa90345f69fa494eb8dbcde852d342e8"
format_stamp: "Formatted at 2026-09-27 15:16:23 on dist-test-slave-f5h6"
server_key: "88fa5f86303d37147141adbad5037929"
server_key_iv: "99e396676dafa35cc9c2d001f343615a"
server_key_version: "encryptionkey@0"
I20260927 15:16:23.227461 24324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:16:23.231068 24339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:23.232530 24324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:16:23.232707 24324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "fa90345f69fa494eb8dbcde852d342e8"
format_stamp: "Formatted at 2026-09-27 15:16:23 on dist-test-slave-f5h6"
server_key: "88fa5f86303d37147141adbad5037929"
server_key_iv: "99e396676dafa35cc9c2d001f343615a"
server_key_version: "encryptionkey@0"
I20260927 15:16:23.232910 24324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:23.254164 24324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:23.258471 24324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:23.258762 24324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:23.259692 24324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:23.261025 24324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:23.261106 24324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:23.261224 24324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:23.261276 24324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:23.290542 24324 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:45281
I20260927 15:16:23.290609 24451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:45281 every 8 connection(s)
I20260927 15:16:23.291805 24324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:16:23.292853 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 24324
I20260927 15:16:23.293041 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:16:23.293684 10373 external_mini_cluster.cc:1428] Setting key a2d075ac1a171d3e5b6b8790ff295303
Sep 27 15:16:23 dist-test-slave-f5h6 krb5kdc[24218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522183, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:16:23.315546 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:23.294612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:42493 (local address 127.10.33.126:35165)
0927 15:16:23.295129 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:23.295136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:23.296036 (+ 900us) server_negotiation.cc:407] Connection header received
0927 15:16:23.298496 (+ 2460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:23.298501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:23.298569 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:23.298674 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:23.300669 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:23.302123 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:23.304427 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:23.304886 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:23.305016 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:23.309010 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:23.309016 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:23.309019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:23.309056 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:23.311946 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:23.312905 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:23.312910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:23.312913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:23.313075 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:23.313594 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:23.313598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:23.313601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:23.313950 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:23.314108 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:23.315040 (+ 932us) server_negotiation.cc:299] Negotiation successful
0927 15:16:23.315243 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":218,"threads_started":1}
I20260927 15:16:23.317111 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:23.294991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.65:42493)
0927 15:16:23.295875 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:23.295928 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:23.298207 (+ 2279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:23.298931 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:23.298942 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:23.299658 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:23.300447 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:23.300472 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:23.302337 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:23.302343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:23.304241 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:23.304260 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:23.305135 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:23.305140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:23.305375 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:23.306203 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:23.306234 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:23.308725 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:23.312125 (+ 3400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:23.312136 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:23.312156 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:23.312759 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:23.313194 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:23.313199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:23.313203 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:23.313453 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:23.314092 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:23.314102 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:23.314592 (+ 490us) client_negotiation.cc:769] Sending connection context
0927 15:16:23.314936 (+ 344us) client_negotiation.cc:240] Negotiation successful
0927 15:16:23.315608 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":294,"threads_started":1}
WARNING: no policy specified for kudu/127.10.33.66@KRBTEST.COM; defaulting to no policy
I20260927 15:16:23.318797 24452 heartbeater.cc:347] Connected to a master server at 127.10.33.126:35165
I20260927 15:16:23.319108 24452 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:23.319878 24452 heartbeater.cc:511] Master 127.10.33.126:35165 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.66@KRBTEST.COM" created.
I20260927 15:16:23.322412 24263 ts_manager.cc:194] Registered new tserver with Master: fa90345f69fa494eb8dbcde852d342e8 (127.10.33.65:45281)
I20260927 15:16:23.324323 24263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:42493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
I20260927 15:16:23.349291 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:35165
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:23.673554 24458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:23.673898 24458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:23.674031 24458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:23.682603 24458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:23.682739 24458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:23.682904 24458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:16:23.692960 24458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:35165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:23.694392 24458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:23.695732 24458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:23.708983 24463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:23.708948 24464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:23.708910 24466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:23.710111 24458 server_base.cc:1034] running on GCE node
I20260927 15:16:23.710783 24458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:23.712280 24458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:23.713590 24458 hybrid_clock.cc:648] HybridClock initialized: now 1790522183713491 us; error 46 us; skew 500 ppm
I20260927 15:16:23.714003 24458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:23 dist-test-slave-f5h6 krb5kdc[24218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522183, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:23.717635 24458 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:16:23.719388 24458 webserver.cc:460] Webserver started at http://127.10.33.66:46595/ using document root <none> and password file <none>
I20260927 15:16:23.720105 24458 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:23.720243 24458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:23.720543 24458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:23.723666 24458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "c5568bb7d7144694bb0eb96ea72d21fd"
format_stamp: "Formatted at 2026-09-27 15:16:23 on dist-test-slave-f5h6"
server_key: "3b3c04363c29f595821a1c5e7bd2c0bb"
server_key_iv: "ab1c47ce8e9bc2ae05cb08301861ef18"
server_key_version: "encryptionkey@0"
I20260927 15:16:23.724495 24458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "c5568bb7d7144694bb0eb96ea72d21fd"
format_stamp: "Formatted at 2026-09-27 15:16:23 on dist-test-slave-f5h6"
server_key: "3b3c04363c29f595821a1c5e7bd2c0bb"
server_key_iv: "ab1c47ce8e9bc2ae05cb08301861ef18"
server_key_version: "encryptionkey@0"
I20260927 15:16:23.730257 24458 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 15:16:23.734230 24473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:23.735767 24458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:16:23.735944 24458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "c5568bb7d7144694bb0eb96ea72d21fd"
format_stamp: "Formatted at 2026-09-27 15:16:23 on dist-test-slave-f5h6"
server_key: "3b3c04363c29f595821a1c5e7bd2c0bb"
server_key_iv: "ab1c47ce8e9bc2ae05cb08301861ef18"
server_key_version: "encryptionkey@0"
I20260927 15:16:23.736142 24458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:23.751709 24458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:23.755894 24458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:23.756170 24458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:23.757119 24458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:23.758514 24458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:23.758600 24458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:23.758733 24458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:23.758785 24458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:23.790349 24458 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:43741
I20260927 15:16:23.790398 24585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:43741 every 8 connection(s)
I20260927 15:16:23.791654 24458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:16:23.801477 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 24458
I20260927 15:16:23.801636 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:16:23.802141 10373 external_mini_cluster.cc:1428] Setting key 11162e1c1603dfbfa830367451f8ea91
Sep 27 15:16:23 dist-test-slave-f5h6 krb5kdc[24218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522183, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:16:23.812168 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:23.794309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:35665 (local address 127.10.33.126:35165)
0927 15:16:23.794552 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:23.794559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:23.795716 (+ 1157us) server_negotiation.cc:407] Connection header received
0927 15:16:23.797564 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:23.797569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:23.797657 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:23.797762 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:23.799307 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:23.800101 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:23.801809 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:23.802318 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:23.802445 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:23.806427 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:23.806434 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:23.806437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:23.806473 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:23.808955 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:23.809767 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:23.809771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:23.809774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:23.809954 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:23.810402 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:23.810407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:23.810410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:23.810688 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:23.810848 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:23.811579 (+ 731us) server_negotiation.cc:299] Negotiation successful
0927 15:16:23.811837 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 15:16:23.813374 24588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:23.794788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.66:35665)
0927 15:16:23.795552 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:23.795594 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:23.797211 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:23.798008 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:23.798017 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:23.798559 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:23.799134 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:23.799152 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:23.800271 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:23.800276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:23.801654 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:23.801670 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:23.802559 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:23.802566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:23.802883 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:23.803702 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:23.803729 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:23.806175 (+ 2446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:23.809131 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:23.809142 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:23.809162 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:23.809626 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:23.810065 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:23.810071 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:23.810075 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:23.810289 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:23.810828 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:23.810838 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:23.811207 (+ 369us) client_negotiation.cc:769] Sending connection context
0927 15:16:23.811466 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 15:16:23.812047 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":313,"threads_started":1}
I20260927 15:16:23.815034 24586 heartbeater.cc:347] Connected to a master server at 127.10.33.126:35165
I20260927 15:16:23.815395 24586 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:23.816322 24586 heartbeater.cc:511] Master 127.10.33.126:35165 requested a full tablet report, sending...
I20260927 15:16:23.818064 24263 ts_manager.cc:194] Registered new tserver with Master: c5568bb7d7144694bb0eb96ea72d21fd (127.10.33.66:43741)
I20260927 15:16:23.819236 24263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:35665
WARNING: no policy specified for kudu/127.10.33.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
I20260927 15:16:23.853152 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:35165
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:24.177845 24591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:24.178200 24591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:24.178329 24591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:24.185990 24591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:24.186125 24591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:24.186261 24591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:16:24.196266 24591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:35165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:24.197571 24591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:24.199156 24591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:24.212270 24599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:24.212226 24597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:24.212437 24596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:24.213665 24591 server_base.cc:1034] running on GCE node
I20260927 15:16:24.214349 24591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:24.215382 24591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:24.216605 24591 hybrid_clock.cc:648] HybridClock initialized: now 1790522184216542 us; error 57 us; skew 500 ppm
I20260927 15:16:24.216985 24591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:24 dist-test-slave-f5h6 krb5kdc[24218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522184, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:24.220180 24591 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:16:24.221714 24591 webserver.cc:460] Webserver started at http://127.10.33.67:43829/ using document root <none> and password file <none>
I20260927 15:16:24.222530 24591 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:24.222654 24591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:24.222940 24591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:24.225806 24591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "713756811bba484baf1c6147e0f9da93"
format_stamp: "Formatted at 2026-09-27 15:16:24 on dist-test-slave-f5h6"
server_key: "3d986d924d31c1b11cbe10bd48372eb6"
server_key_iv: "741048f6c45c3b10e3b6dfd6a99ab352"
server_key_version: "encryptionkey@0"
I20260927 15:16:24.226621 24591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "713756811bba484baf1c6147e0f9da93"
format_stamp: "Formatted at 2026-09-27 15:16:24 on dist-test-slave-f5h6"
server_key: "3d986d924d31c1b11cbe10bd48372eb6"
server_key_iv: "741048f6c45c3b10e3b6dfd6a99ab352"
server_key_version: "encryptionkey@0"
I20260927 15:16:24.231801 24591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:16:24.235504 24606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:24.237174 24591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:16:24.237349 24591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "713756811bba484baf1c6147e0f9da93"
format_stamp: "Formatted at 2026-09-27 15:16:24 on dist-test-slave-f5h6"
server_key: "3d986d924d31c1b11cbe10bd48372eb6"
server_key_iv: "741048f6c45c3b10e3b6dfd6a99ab352"
server_key_version: "encryptionkey@0"
I20260927 15:16:24.237545 24591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:24.264621 24591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:24.268731 24591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:24.269001 24591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:24.269989 24591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:24.271677 24591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:24.271765 24591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:24.271874 24591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:24.271924 24591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:24.303414 24591 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:43765
I20260927 15:16:24.303484 24718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:43765 every 8 connection(s)
I20260927 15:16:24.304690 24591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:16:24.305516 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 24591
I20260927 15:16:24.305714 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:16:24.306290 10373 external_mini_cluster.cc:1428] Setting key 17b247b8671beb9b36943a97621d049c
Sep 27 15:16:24 dist-test-slave-f5h6 krb5kdc[24218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522184, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:16:24.326645 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:24.307459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:43431 (local address 127.10.33.126:35165)
0927 15:16:24.307640 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:24.307647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:24.308961 (+ 1314us) server_negotiation.cc:407] Connection header received
0927 15:16:24.310861 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:24.310865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:24.310932 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:24.311014 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:24.312570 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.313376 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.315311 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.315763 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.315880 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:24.319815 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:24.319821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:24.319825 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:24.319866 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:24.323379 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:24.324200 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:24.324204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:24.324206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:24.324337 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:24.324927 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:24.324931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:24.324933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:24.325217 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:24.325328 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:24.326215 (+ 887us) server_negotiation.cc:299] Negotiation successful
0927 15:16:24.326399 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:16:24.327661 24721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:24.307939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.67:43431)
0927 15:16:24.308791 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:24.308838 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:24.310610 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:24.311273 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:24.311282 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:24.311827 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:24.312384 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:24.312403 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.313548 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.313552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:24.315083 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:24.315108 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.316008 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.316015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:24.316247 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:24.316956 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:24.316981 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:24.319512 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:24.323548 (+ 4036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:24.323556 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:24.323573 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:24.324074 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:24.324464 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:24.324468 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:24.324471 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:24.324798 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:24.325340 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:24.325348 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:24.325749 (+ 401us) client_negotiation.cc:769] Sending connection context
0927 15:16:24.326077 (+ 328us) client_negotiation.cc:240] Negotiation successful
0927 15:16:24.326511 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":291,"threads_started":1}
I20260927 15:16:24.328243 24452 heartbeater.cc:503] Master 127.10.33.126:35165 was elected leader, sending a full tablet report...
I20260927 15:16:24.329303 24719 heartbeater.cc:347] Connected to a master server at 127.10.33.126:35165
I20260927 15:16:24.329639 24719 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:24.330446 24719 heartbeater.cc:511] Master 127.10.33.126:35165 requested a full tablet report, sending...
I20260927 15:16:24.331961 24263 ts_manager.cc:194] Registered new tserver with Master: 713756811bba484baf1c6147e0f9da93 (127.10.33.67:43765)
I20260927 15:16:24.332798 24263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:43431
I20260927 15:16:24.334998 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:16:24.352368 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:24.339778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44252 (local address 127.10.33.126:35165)
0927 15:16:24.339995 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:24.340002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:24.340345 (+ 343us) server_negotiation.cc:407] Connection header received
0927 15:16:24.340535 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:24.340539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:24.340602 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:24.340685 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:24.342194 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.342917 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.344285 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.344738 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.344857 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:24.347311 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:24.347317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:24.347320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:24.347355 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:24.349769 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:24.350430 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:24.350436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:24.350440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:24.350568 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:24.350923 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:24.350927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:24.350929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:24.351273 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:24.351509 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:24.351824 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 15:16:24.352067 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:16:24.356398 24263 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:16:24.363271 24263 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:16:24.387698 24733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:24.375735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41180 (local address 127.10.33.65:45281)
0927 15:16:24.377844 (+ 2109us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:24.377873 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:24.377903 (+ 30us) server_negotiation.cc:407] Connection header received
0927 15:16:24.378001 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:24.378009 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:24.378288 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:24.379118 (+ 830us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:24.380586 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.382307 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.385068 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.386581 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.386706 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:24.386939 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:24.387183 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 15:16:24.387373 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1946,"thread_start_us":1791,"threads_started":1}
I20260927 15:16:24.387775 24731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:24.375288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:45281 (local address 127.0.0.1:41180)
0927 15:16:24.376161 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:24.376203 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:24.376371 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:24.378541 (+ 2170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:24.378562 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:24.378596 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:24.379128 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:24.379140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.382475 (+ 3335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.382481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:24.384878 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:24.384897 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.386762 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.386768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:24.386963 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:24.387011 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 15:16:24.387122 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 15:16:24.387359 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":25,"thread_start_us":570,"threads_started":1}
I20260927 15:16:24.388733 24734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:24.375982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.10.33.66:43741)
0927 15:16:24.377123 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:24.377131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:24.378059 (+ 928us) server_negotiation.cc:407] Connection header received
0927 15:16:24.378167 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:24.378174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:24.378395 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:24.378592 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:24.379802 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.381388 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.386003 (+ 4615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.387336 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.387463 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:24.387655 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:24.388200 (+ 545us) server_negotiation.cc:299] Negotiation successful
0927 15:16:24.388370 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":887,"thread_start_us":1191,"threads_started":1}
I20260927 15:16:24.389848 24732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:24.375463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:43741 (local address 127.0.0.1:54432)
0927 15:16:24.377777 (+ 2314us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:24.377792 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:24.377922 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:24.378930 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:24.378940 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:24.378968 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:24.379613 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:24.379623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.381561 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.381567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:24.385828 (+ 4261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:24.385843 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.387807 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.387813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:24.387987 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:24.388020 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:16:24.389297 (+ 1277us) client_negotiation.cc:240] Negotiation successful
0927 15:16:24.389446 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2196,"mutex_wait_us":1,"thread_start_us":2071,"threads_started":1}
I20260927 15:16:24.392359 24387 tablet_service.cc:1467] Processing CreateTablet for tablet 52891f88cfe44f18b7e18b508a76114a (DEFAULT_TABLE table=test-table [id=2cd0d8207af74b93b3006fd2c8d249f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:24.393061 24735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:24.377958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:43765 (local address 127.0.0.1:38174)
0927 15:16:24.384748 (+ 6790us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:24.384764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:24.384854 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:24.386025 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:24.386030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:24.386057 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:24.386393 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:24.386404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.388261 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.388265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:24.390488 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:24.390506 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.392285 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.392292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:24.392463 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:24.392497 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 15:16:24.392597 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 15:16:24.392737 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6674,"thread_start_us":6515,"threads_started":1}
I20260927 15:16:24.393113 24736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:24.378222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38174 (local address 127.10.33.67:43765)
0927 15:16:24.385502 (+ 7280us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:24.385508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:24.385537 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:16:24.385613 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:24.385619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:24.385822 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:24.386046 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:24.386568 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.388076 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.390673 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.392059 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.392211 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:24.392445 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:24.392654 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 15:16:24.392820 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7124,"thread_start_us":6984,"threads_started":1}
I20260927 15:16:24.393290 24521 tablet_service.cc:1467] Processing CreateTablet for tablet 52891f88cfe44f18b7e18b508a76114a (DEFAULT_TABLE table=test-table [id=2cd0d8207af74b93b3006fd2c8d249f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:24.394228 24387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52891f88cfe44f18b7e18b508a76114a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:24.394605 24521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52891f88cfe44f18b7e18b508a76114a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:24.397255 24654 tablet_service.cc:1467] Processing CreateTablet for tablet 52891f88cfe44f18b7e18b508a76114a (DEFAULT_TABLE table=test-table [id=2cd0d8207af74b93b3006fd2c8d249f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:24.398635 24654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52891f88cfe44f18b7e18b508a76114a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:24.405431 24737 tablet_bootstrap.cc:492] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8: Bootstrap starting.
I20260927 15:16:24.408984 24737 tablet_bootstrap.cc:654] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:24.410030 24737 log.cc:824] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:24.411413 24738 tablet_bootstrap.cc:492] T 52891f88cfe44f18b7e18b508a76114a P c5568bb7d7144694bb0eb96ea72d21fd: Bootstrap starting.
I20260927 15:16:24.413179 24737 tablet_bootstrap.cc:492] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8: No bootstrap required, opened a new log
I20260927 15:16:24.413465 24737 ts_tablet_manager.cc:1397] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260927 15:16:24.414187 24740 tablet_bootstrap.cc:492] T 52891f88cfe44f18b7e18b508a76114a P 713756811bba484baf1c6147e0f9da93: Bootstrap starting.
I20260927 15:16:24.415129 24738 tablet_bootstrap.cc:654] T 52891f88cfe44f18b7e18b508a76114a P c5568bb7d7144694bb0eb96ea72d21fd: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:24.416373 24738 log.cc:824] T 52891f88cfe44f18b7e18b508a76114a P c5568bb7d7144694bb0eb96ea72d21fd: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:24.417654 24740 tablet_bootstrap.cc:654] T 52891f88cfe44f18b7e18b508a76114a P 713756811bba484baf1c6147e0f9da93: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:24.418735 24740 log.cc:824] T 52891f88cfe44f18b7e18b508a76114a P 713756811bba484baf1c6147e0f9da93: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:24.421661 24738 tablet_bootstrap.cc:492] T 52891f88cfe44f18b7e18b508a76114a P c5568bb7d7144694bb0eb96ea72d21fd: No bootstrap required, opened a new log
I20260927 15:16:24.421792 24740 tablet_bootstrap.cc:492] T 52891f88cfe44f18b7e18b508a76114a P 713756811bba484baf1c6147e0f9da93: No bootstrap required, opened a new log
I20260927 15:16:24.421986 24738 ts_tablet_manager.cc:1397] T 52891f88cfe44f18b7e18b508a76114a P c5568bb7d7144694bb0eb96ea72d21fd: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.004s
I20260927 15:16:24.422091 24740 ts_tablet_manager.cc:1397] T 52891f88cfe44f18b7e18b508a76114a P 713756811bba484baf1c6147e0f9da93: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 15:16:24.424157 24737 raft_consensus.cc:348] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa90345f69fa494eb8dbcde852d342e8" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45281 } } peers { permanent_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 43741 } } peers { permanent_uuid: "713756811bba484baf1c6147e0f9da93" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43765 } }
I20260927 15:16:24.424676 24737 raft_consensus.cc:374] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:24.424791 24737 raft_consensus.cc:729] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa90345f69fa494eb8dbcde852d342e8, State: Initialized, Role: FOLLOWER
I20260927 15:16:24.425451 24737 consensus_queue.cc:260] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa90345f69fa494eb8dbcde852d342e8" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45281 } } peers { permanent_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 43741 } } peers { permanent_uuid: "713756811bba484baf1c6147e0f9da93" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43765 } }
I20260927 15:16:24.429759 24737 ts_tablet_manager.cc:1428] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260927 15:16:24.430425 24738 raft_consensus.cc:348] T 52891f88cfe44f18b7e18b508a76114a P c5568bb7d7144694bb0eb96ea72d21fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa90345f69fa494eb8dbcde852d342e8" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45281 } } peers { permanent_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 43741 } } peers { permanent_uuid: "713756811bba484baf1c6147e0f9da93" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43765 } }
I20260927 15:16:24.430922 24738 raft_consensus.cc:374] T 52891f88cfe44f18b7e18b508a76114a P c5568bb7d7144694bb0eb96ea72d21fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:24.431043 24738 raft_consensus.cc:729] T 52891f88cfe44f18b7e18b508a76114a P c5568bb7d7144694bb0eb96ea72d21fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5568bb7d7144694bb0eb96ea72d21fd, State: Initialized, Role: FOLLOWER
I20260927 15:16:24.431654 24738 consensus_queue.cc:260] T 52891f88cfe44f18b7e18b508a76114a P c5568bb7d7144694bb0eb96ea72d21fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa90345f69fa494eb8dbcde852d342e8" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45281 } } peers { permanent_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 43741 } } peers { permanent_uuid: "713756811bba484baf1c6147e0f9da93" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43765 } }
I20260927 15:16:24.432252 24740 raft_consensus.cc:348] T 52891f88cfe44f18b7e18b508a76114a P 713756811bba484baf1c6147e0f9da93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa90345f69fa494eb8dbcde852d342e8" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45281 } } peers { permanent_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 43741 } } peers { permanent_uuid: "713756811bba484baf1c6147e0f9da93" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43765 } }
I20260927 15:16:24.432713 24740 raft_consensus.cc:374] T 52891f88cfe44f18b7e18b508a76114a P 713756811bba484baf1c6147e0f9da93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:24.432827 24740 raft_consensus.cc:729] T 52891f88cfe44f18b7e18b508a76114a P 713756811bba484baf1c6147e0f9da93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 713756811bba484baf1c6147e0f9da93, State: Initialized, Role: FOLLOWER
I20260927 15:16:24.433446 24740 consensus_queue.cc:260] T 52891f88cfe44f18b7e18b508a76114a P 713756811bba484baf1c6147e0f9da93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa90345f69fa494eb8dbcde852d342e8" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45281 } } peers { permanent_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 43741 } } peers { permanent_uuid: "713756811bba484baf1c6147e0f9da93" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43765 } }
I20260927 15:16:24.434567 24719 heartbeater.cc:503] Master 127.10.33.126:35165 was elected leader, sending a full tablet report...
I20260927 15:16:24.437018 24740 ts_tablet_manager.cc:1428] T 52891f88cfe44f18b7e18b508a76114a P 713756811bba484baf1c6147e0f9da93: Time spent starting tablet: real 0.015s user 0.010s sys 0.002s
I20260927 15:16:24.437978 24586 heartbeater.cc:503] Master 127.10.33.126:35165 was elected leader, sending a full tablet report...
I20260927 15:16:24.438774 24738 ts_tablet_manager.cc:1428] T 52891f88cfe44f18b7e18b508a76114a P c5568bb7d7144694bb0eb96ea72d21fd: Time spent starting tablet: real 0.017s user 0.015s sys 0.001s
W20260927 15:16:24.543538 24587 tablet.cc:2369] T 52891f88cfe44f18b7e18b508a76114a P c5568bb7d7144694bb0eb96ea72d21fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:16:24.544101 24453 tablet.cc:2369] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:16:24.556463 24720 tablet.cc:2369] T 52891f88cfe44f18b7e18b508a76114a P 713756811bba484baf1c6147e0f9da93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:16:24.687913 24743 raft_consensus.cc:482] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:16:24.688213 24743 raft_consensus.cc:504] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa90345f69fa494eb8dbcde852d342e8" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45281 } } peers { permanent_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 43741 } } peers { permanent_uuid: "713756811bba484baf1c6147e0f9da93" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43765 } }
I20260927 15:16:24.690016 24743 leader_election.cc:288] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5568bb7d7144694bb0eb96ea72d21fd (127.10.33.66:43741), 713756811bba484baf1c6147e0f9da93 (127.10.33.67:43765)
I20260927 15:16:24.696269 24734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:24.690376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:59529 (local address 127.10.33.66:43741)
0927 15:16:24.690563 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:24.690568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:24.691165 (+ 597us) server_negotiation.cc:407] Connection header received
0927 15:16:24.691405 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:24.691412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:24.691498 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:24.691592 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:24.692168 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.692867 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.694637 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.695397 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.695544 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:24.695621 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:24.695893 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 15:16:24.696017 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:16:24.696383 24746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:24.690376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:43741 (local address 127.10.33.65:59529)
0927 15:16:24.691026 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:24.691041 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:24.691223 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:24.691684 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:24.691689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:24.691717 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:24.692030 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:24.692038 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.692991 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.692997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:24.694483 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:24.694495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.695598 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.695605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:24.695745 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:24.695768 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:16:24.695848 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 15:16:24.695979 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":3,"thread_start_us":361,"threads_started":1}
I20260927 15:16:24.696733 24736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:24.691468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:51927 (local address 127.10.33.67:43765)
0927 15:16:24.691672 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:24.691677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:24.691870 (+ 193us) server_negotiation.cc:407] Connection header received
0927 15:16:24.692036 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:24.692041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:24.692101 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:24.692173 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:24.692654 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.693367 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.695077 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.695857 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.696003 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:24.696073 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:24.696365 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 15:16:24.696492 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:16:24.696774 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:24.691171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:43765 (local address 127.10.33.65:51927)
0927 15:16:24.691782 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:24.691798 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:24.691914 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:24.692212 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:24.692217 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:24.692239 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:24.692515 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:24.692524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.693509 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.693514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:24.694925 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:24.694937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.696070 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.696076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:24.696223 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:24.696245 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:16:24.696324 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 15:16:24.696450 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":287,"threads_started":1}
I20260927 15:16:24.697533 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52891f88cfe44f18b7e18b508a76114a" candidate_uuid: "fa90345f69fa494eb8dbcde852d342e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" is_pre_election: true
I20260927 15:16:24.697661 24674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52891f88cfe44f18b7e18b508a76114a" candidate_uuid: "fa90345f69fa494eb8dbcde852d342e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "713756811bba484baf1c6147e0f9da93" is_pre_election: true
I20260927 15:16:24.698170 24541 raft_consensus.cc:2454] T 52891f88cfe44f18b7e18b508a76114a P c5568bb7d7144694bb0eb96ea72d21fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa90345f69fa494eb8dbcde852d342e8 in term 0.
I20260927 15:16:24.698169 24674 raft_consensus.cc:2454] T 52891f88cfe44f18b7e18b508a76114a P 713756811bba484baf1c6147e0f9da93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa90345f69fa494eb8dbcde852d342e8 in term 0.
I20260927 15:16:24.698827 24340 leader_election.cc:302] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 713756811bba484baf1c6147e0f9da93, fa90345f69fa494eb8dbcde852d342e8; no voters:
I20260927 15:16:24.699188 24743 raft_consensus.cc:2781] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:16:24.699334 24743 raft_consensus.cc:482] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:16:24.699433 24743 raft_consensus.cc:3037] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:24.701694 24743 raft_consensus.cc:504] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa90345f69fa494eb8dbcde852d342e8" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45281 } } peers { permanent_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 43741 } } peers { permanent_uuid: "713756811bba484baf1c6147e0f9da93" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43765 } }
I20260927 15:16:24.702294 24743 leader_election.cc:288] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8 [CANDIDATE]: Term 1 election: Requested vote from peers c5568bb7d7144694bb0eb96ea72d21fd (127.10.33.66:43741), 713756811bba484baf1c6147e0f9da93 (127.10.33.67:43765)
I20260927 15:16:24.702924 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52891f88cfe44f18b7e18b508a76114a" candidate_uuid: "fa90345f69fa494eb8dbcde852d342e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5568bb7d7144694bb0eb96ea72d21fd"
I20260927 15:16:24.703065 24674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52891f88cfe44f18b7e18b508a76114a" candidate_uuid: "fa90345f69fa494eb8dbcde852d342e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "713756811bba484baf1c6147e0f9da93"
I20260927 15:16:24.703192 24541 raft_consensus.cc:3037] T 52891f88cfe44f18b7e18b508a76114a P c5568bb7d7144694bb0eb96ea72d21fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:24.703292 24674 raft_consensus.cc:3037] T 52891f88cfe44f18b7e18b508a76114a P 713756811bba484baf1c6147e0f9da93 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:24.706633 24674 raft_consensus.cc:2454] T 52891f88cfe44f18b7e18b508a76114a P 713756811bba484baf1c6147e0f9da93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa90345f69fa494eb8dbcde852d342e8 in term 1.
I20260927 15:16:24.706633 24541 raft_consensus.cc:2454] T 52891f88cfe44f18b7e18b508a76114a P c5568bb7d7144694bb0eb96ea72d21fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa90345f69fa494eb8dbcde852d342e8 in term 1.
I20260927 15:16:24.707141 24340 leader_election.cc:302] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5568bb7d7144694bb0eb96ea72d21fd, fa90345f69fa494eb8dbcde852d342e8; no voters:
I20260927 15:16:24.707443 24743 raft_consensus.cc:2781] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:24.707896 24743 raft_consensus.cc:686] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8 [term 1 LEADER]: Becoming Leader. State: Replica: fa90345f69fa494eb8dbcde852d342e8, State: Running, Role: LEADER
I20260927 15:16:24.708364 24743 consensus_queue.cc:237] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa90345f69fa494eb8dbcde852d342e8" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45281 } } peers { permanent_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 43741 } } peers { permanent_uuid: "713756811bba484baf1c6147e0f9da93" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43765 } }
I20260927 15:16:24.713269 24262 catalog_manager.cc:5495] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa90345f69fa494eb8dbcde852d342e8 (127.10.33.65). New cstate: current_term: 1 leader_uuid: "fa90345f69fa494eb8dbcde852d342e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa90345f69fa494eb8dbcde852d342e8" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 43741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "713756811bba484baf1c6147e0f9da93" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43765 } health_report { overall_health: UNKNOWN } } }
I20260927 15:16:24.755988 24733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:24.750658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.10.33.65:45281)
0927 15:16:24.750857 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:24.750862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:24.750880 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:16:24.751000 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:24.751004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:24.751108 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:24.751267 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:16:24.751613 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.752395 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.753811 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.754323 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.754431 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:24.754958 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:16:24.755482 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:16:24.755596 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:24.755711 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 15:16:24.755780 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 15:16:24.764041 24541 raft_consensus.cc:1261] T 52891f88cfe44f18b7e18b508a76114a P c5568bb7d7144694bb0eb96ea72d21fd [term 1 FOLLOWER]: Refusing update from remote peer fa90345f69fa494eb8dbcde852d342e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:24.764042 24674 raft_consensus.cc:1261] T 52891f88cfe44f18b7e18b508a76114a P 713756811bba484baf1c6147e0f9da93 [term 1 FOLLOWER]: Refusing update from remote peer fa90345f69fa494eb8dbcde852d342e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:24.765041 24748 consensus_queue.cc:1035] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713756811bba484baf1c6147e0f9da93" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:24.765375 24743 consensus_queue.cc:1035] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 43741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:24.779655 24753 mvcc.cc:205] Tried to move back new op lower bound from 7333978868783345664 to 7333978868569124864. Current Snapshot: MvccSnapshot[applied={T|T < 7333978868783345664}]
I20260927 15:16:24.780339 24752 mvcc.cc:205] Tried to move back new op lower bound from 7333978868783345664 to 7333978868569124864. Current Snapshot: MvccSnapshot[applied={T|T < 7333978868783345664}]
I20260927 15:16:24.789826 24263 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:24.790153 24263 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:24.793601 24263 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295: Sending DeleteTablet for 3 replicas of tablet 52891f88cfe44f18b7e18b508a76114a
I20260927 15:16:24.794809 24387 tablet_service.cc:1514] Processing DeleteTablet for tablet 52891f88cfe44f18b7e18b508a76114a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:24 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:41180
I20260927 15:16:24.795686 24760 tablet_replica.cc:330] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8: stopping tablet replica
I20260927 15:16:24.796187 24760 raft_consensus.cc:2229] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:16:24.796252 24521 tablet_service.cc:1514] Processing DeleteTablet for tablet 52891f88cfe44f18b7e18b508a76114a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:24 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:54432
I20260927 15:16:24.796911 24760 raft_consensus.cc:2258] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:24.797318 24654 tablet_service.cc:1514] Processing DeleteTablet for tablet 52891f88cfe44f18b7e18b508a76114a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:24 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:38174
I20260927 15:16:24.797775 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 24324
I20260927 15:16:24.806142 24760 ts_tablet_manager.cc:1905] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:24.817204 24760 ts_tablet_manager.cc:1918] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:16:24.817533 24760 log.cc:1196] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/52891f88cfe44f18b7e18b508a76114a
I20260927 15:16:24.818305 24760 ts_tablet_manager.cc:1939] T 52891f88cfe44f18b7e18b508a76114a P fa90345f69fa494eb8dbcde852d342e8: Deleting consensus metadata
I20260927 15:16:24.820374 24249 catalog_manager.cc:4841] TS fa90345f69fa494eb8dbcde852d342e8 (127.10.33.65:45281): tablet 52891f88cfe44f18b7e18b508a76114a (table test-table [id=2cd0d8207af74b93b3006fd2c8d249f6]) successfully deleted
I20260927 15:16:24.821308 24761 tablet_replica.cc:330] T 52891f88cfe44f18b7e18b508a76114a P c5568bb7d7144694bb0eb96ea72d21fd: stopping tablet replica
I20260927 15:16:24.822362 24761 raft_consensus.cc:2229] T 52891f88cfe44f18b7e18b508a76114a P c5568bb7d7144694bb0eb96ea72d21fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:16:24.823076 24761 raft_consensus.cc:2258] T 52891f88cfe44f18b7e18b508a76114a P c5568bb7d7144694bb0eb96ea72d21fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:24.823985 24762 tablet_replica.cc:330] T 52891f88cfe44f18b7e18b508a76114a P 713756811bba484baf1c6147e0f9da93: stopping tablet replica
I20260927 15:16:24.824682 24762 raft_consensus.cc:2229] T 52891f88cfe44f18b7e18b508a76114a P 713756811bba484baf1c6147e0f9da93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:16:24.825259 24762 raft_consensus.cc:2258] T 52891f88cfe44f18b7e18b508a76114a P 713756811bba484baf1c6147e0f9da93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:24.826331 24761 ts_tablet_manager.cc:1905] T 52891f88cfe44f18b7e18b508a76114a P c5568bb7d7144694bb0eb96ea72d21fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:24.827777 24762 ts_tablet_manager.cc:1905] T 52891f88cfe44f18b7e18b508a76114a P 713756811bba484baf1c6147e0f9da93: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 15:16:24 dist-test-slave-f5h6 krb5kdc[24218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.65@KRBTEST.COM
I20260927 15:16:24.833879 24761 ts_tablet_manager.cc:1918] T 52891f88cfe44f18b7e18b508a76114a P c5568bb7d7144694bb0eb96ea72d21fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:16:24.834121 24761 log.cc:1196] T 52891f88cfe44f18b7e18b508a76114a P c5568bb7d7144694bb0eb96ea72d21fd: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/52891f88cfe44f18b7e18b508a76114a
I20260927 15:16:24.834733 24761 ts_tablet_manager.cc:1939] T 52891f88cfe44f18b7e18b508a76114a P c5568bb7d7144694bb0eb96ea72d21fd: Deleting consensus metadata
I20260927 15:16:24.835171 24762 ts_tablet_manager.cc:1918] T 52891f88cfe44f18b7e18b508a76114a P 713756811bba484baf1c6147e0f9da93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:16:24.835407 24762 log.cc:1196] T 52891f88cfe44f18b7e18b508a76114a P 713756811bba484baf1c6147e0f9da93: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/52891f88cfe44f18b7e18b508a76114a
I20260927 15:16:24.835981 24762 ts_tablet_manager.cc:1939] T 52891f88cfe44f18b7e18b508a76114a P 713756811bba484baf1c6147e0f9da93: Deleting consensus metadata
I20260927 15:16:24.836966 24247 catalog_manager.cc:4841] TS c5568bb7d7144694bb0eb96ea72d21fd (127.10.33.66:43741): tablet 52891f88cfe44f18b7e18b508a76114a (table test-table [id=2cd0d8207af74b93b3006fd2c8d249f6]) successfully deleted
I20260927 15:16:24.838650 24247 catalog_manager.cc:4841] TS 713756811bba484baf1c6147e0f9da93 (127.10.33.67:43765): tablet 52891f88cfe44f18b7e18b508a76114a (table test-table [id=2cd0d8207af74b93b3006fd2c8d249f6]) successfully deleted
I20260927 15:16:24.842988 24733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:24.799613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41206 (local address 127.10.33.65:45281)
0927 15:16:24.799872 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:24.799880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:24.822063 (+ 22183us) server_negotiation.cc:407] Connection header received
0927 15:16:24.822376 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:24.822382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:24.822530 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:24.822607 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:24.824613 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.826182 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.828213 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.828884 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.828983 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:24.834608 (+ 5625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:24.834637 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:24.834661 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:24.834713 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:24.839213 (+ 4500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:24.840242 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:24.840251 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:24.840253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:24.840424 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:24.840938 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:24.840943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:24.840945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:24.841440 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:24.841910 (+ 470us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:24.842396 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 15:16:24.842640 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260927 15:16:24.843686 24447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:24.942509 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 24324
I20260927 15:16:24.958523 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 24458
Sep 27 15:16:24 dist-test-slave-f5h6 krb5kdc[24218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.66@KRBTEST.COM
I20260927 15:16:24.973905 24734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:24.959255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54448 (local address 127.10.33.66:43741)
0927 15:16:24.959439 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:24.959444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:24.959462 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:16:24.959603 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:24.959607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:24.959717 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:24.959800 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:24.961126 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.961824 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.963253 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:24.963703 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:24.963812 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:24.968335 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:24.968352 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:24.968369 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:24.968409 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:24.971159 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:24.971915 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:24.971923 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:24.971926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:24.972105 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:24.972452 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:24.972457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:24.972459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:24.972883 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:24.973126 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:24.973410 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 15:16:24.973605 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:16:24.974529 24581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:25.063166 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 24458
I20260927 15:16:25.076679 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 24591
Sep 27 15:16:25 dist-test-slave-f5h6 krb5kdc[24218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.67@KRBTEST.COM
I20260927 15:16:25.092165 24736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:25.077464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38182 (local address 127.10.33.67:43765)
0927 15:16:25.077637 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:25.077642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:25.077661 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:16:25.077836 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:25.077840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:25.077938 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:25.078020 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:25.079394 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:25.080197 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:25.081479 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:25.081911 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:25.082004 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:25.086494 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:25.086511 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:25.086528 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:25.086582 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:25.089511 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:25.090311 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:25.090319 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:25.090321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:25.090479 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:25.090775 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:25.090778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:25.090780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:25.091144 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:25.091370 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:25.091677 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 15:16:25.091845 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:16:25.092790 24714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:25.181186 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 24591
I20260927 15:16:25.194628 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 24231
I20260927 15:16:25.195803 24293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:25.309398 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 24231
I20260927 15:16:25.327426 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:35165
--webserver_interface=127.10.33.126
--webserver_port=41623
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:35165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:25.657823 24768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:25.658205 24768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:25.658344 24768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:25.665794 24768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:25.665951 24768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:25.666023 24768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:25.666101 24768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:25.666160 24768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:16:25.677147 24768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:35165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:35165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=41623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:25.678627 24768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:25.680001 24768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:25.703543 24774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:25.703580 24775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:25.703516 24777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:25.704386 24768 server_base.cc:1034] running on GCE node
I20260927 15:16:25.705235 24768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:25.706916 24768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:25.708156 24768 hybrid_clock.cc:648] HybridClock initialized: now 1790522185708102 us; error 37 us; skew 500 ppm
I20260927 15:16:25.708554 24768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:25 dist-test-slave-f5h6 krb5kdc[24218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522185, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:25.712250 24768 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:16:25.714069 24768 webserver.cc:460] Webserver started at http://127.10.33.126:41623/ using document root <none> and password file <none>
I20260927 15:16:25.714833 24768 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:25.714972 24768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:25.721320 24768 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 15:16:25.724571 24784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:25.726063 24768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 15:16:25.726249 24768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "e5a5cd3ea7144cac9eb3e472f7f51295"
format_stamp: "Formatted at 2026-09-27 15:16:22 on dist-test-slave-f5h6"
server_key: "2e5ed7f594e28d5dcc658b6c680dac79"
server_key_iv: "e8c8e4fa99ecbd10fd1052170a529cb3"
server_key_version: "encryptionkey@0"
I20260927 15:16:25.727391 24768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:25.746791 24768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:25.751206 24768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:25.751471 24768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:25.769209 24768 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:35165
I20260927 15:16:25.769246 24835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:35165 every 8 connection(s)
I20260927 15:16:25.770777 24768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:16:25.777837 24836 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:16:25.780625 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 24768
I20260927 15:16:25.781658 24836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295: Bootstrap starting.
I20260927 15:16:25.781654 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:45281
--local_ip_for_outbound_sockets=127.10.33.65
--tserver_master_addrs=127.10.33.126:35165
--webserver_port=39529
--webserver_interface=127.10.33.65
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:16:25.803746 24836 log.cc:824] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:25.815976 24836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:16:25.816478 24836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295: Bootstrap complete.
I20260927 15:16:25.825506 24836 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35165 } }
I20260927 15:16:25.826378 24836 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5a5cd3ea7144cac9eb3e472f7f51295, State: Initialized, Role: FOLLOWER
I20260927 15:16:25.827006 24836 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35165 } }
I20260927 15:16:25.827252 24836 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:16:25.827373 24836 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:16:25.827565 24836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:16:25.831624 24836 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35165 } }
I20260927 15:16:25.832237 24836 leader_election.cc:302] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5a5cd3ea7144cac9eb3e472f7f51295; no voters:
I20260927 15:16:25.832813 24836 leader_election.cc:288] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:16:25.832957 24840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:16:25.834160 24840 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 2 LEADER]: Becoming Leader. State: Replica: e5a5cd3ea7144cac9eb3e472f7f51295, State: Running, Role: LEADER
I20260927 15:16:25.834851 24840 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35165 } }
I20260927 15:16:25.835198 24836 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:25.837383 24842 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5a5cd3ea7144cac9eb3e472f7f51295. Latest consensus state: current_term: 2 leader_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35165 } } }
I20260927 15:16:25.837461 24841 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35165 } } }
I20260927 15:16:25.837730 24842 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:25.837815 24841 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:25.851357 24845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:16:25.860644 24845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:16:25.862493 24845 catalog_manager.cc:1259] Loaded cluster ID: ed96f4cbbb8840d0b2166907e5a90c55
I20260927 15:16:25.862679 24845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:16:25.866250 24845 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 15:16:25.882404 24845 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 15:16:25.889593 24845 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:16:25.891047 24845 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295: Loaded TSK: 0
I20260927 15:16:25.892381 24845 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:16:26.149547 24838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:26.149889 24838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:26.150069 24838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:26.157815 24838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:26.158015 24838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:26.158180 24838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:16:26.168831 24838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:45281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=39529
--tserver_master_addrs=127.10.33.126:35165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:26.170486 24838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:26.171940 24838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:26.187754 24864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:26.187827 24863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:26.188038 24866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:26.188575 24838 server_base.cc:1034] running on GCE node
I20260927 15:16:26.189254 24838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:26.190325 24838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:26.191552 24838 hybrid_clock.cc:648] HybridClock initialized: now 1790522186191485 us; error 56 us; skew 500 ppm
I20260927 15:16:26.191895 24838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:26 dist-test-slave-f5h6 krb5kdc[24218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522186, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:26.195205 24838 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:16:26.196775 24838 webserver.cc:460] Webserver started at http://127.10.33.65:39529/ using document root <none> and password file <none>
I20260927 15:16:26.197448 24838 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:26.197577 24838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:26.203235 24838 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:16:26.206108 24873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:26.207656 24838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:16:26.207814 24838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "fa90345f69fa494eb8dbcde852d342e8"
format_stamp: "Formatted at 2026-09-27 15:16:23 on dist-test-slave-f5h6"
server_key: "88fa5f86303d37147141adbad5037929"
server_key_iv: "99e396676dafa35cc9c2d001f343615a"
server_key_version: "encryptionkey@0"
I20260927 15:16:26.208786 24838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:26.234007 24838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:26.238927 24838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:26.239202 24838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:26.240248 24838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:26.241732 24838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:26.241838 24838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:26.241988 24838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:26.242132 24838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:26.273165 24838 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:45281
I20260927 15:16:26.273232 24985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:45281 every 8 connection(s)
I20260927 15:16:26.274672 24838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:16:26.281356 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 24838
I20260927 15:16:26.282090 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:43741
--local_ip_for_outbound_sockets=127.10.33.66
--tserver_master_addrs=127.10.33.126:35165
--webserver_port=46595
--webserver_interface=127.10.33.66
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:16:26 dist-test-slave-f5h6 krb5kdc[24218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522186, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:16:26.298363 24988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:26.277421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:59341 (local address 127.10.33.126:35165)
0927 15:16:26.278012 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:26.278023 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:26.278679 (+ 656us) server_negotiation.cc:407] Connection header received
0927 15:16:26.280789 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:26.280802 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:26.281227 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:26.281839 (+ 612us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:26.283429 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:26.284784 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:26.286359 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:26.287001 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:26.287172 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:26.290888 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:26.290895 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:26.290911 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:26.290948 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:26.293745 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:26.294663 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:26.294670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:26.294676 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:26.294869 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:26.295370 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:26.295373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:26.295376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:26.295877 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:26.296034 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:26.296935 (+ 901us) server_negotiation.cc:299] Negotiation successful
0927 15:16:26.297271 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":223,"threads_started":1}
I20260927 15:16:26.298628 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:26.277600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.65:59341)
0927 15:16:26.278505 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:26.278567 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:26.280224 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:26.282001 (+ 1777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:26.282039 (+ 38us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:26.282681 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:26.283208 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:26.283227 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:26.284947 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:26.284951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:26.286220 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:26.286233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:26.287233 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:26.287255 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:26.287495 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:26.288173 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:26.288196 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:26.290566 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:26.293923 (+ 3357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:26.293932 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:26.293945 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:26.294486 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:26.294983 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:26.294988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:26.294991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:26.295221 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:26.296013 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:26.296022 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:26.296433 (+ 411us) client_negotiation.cc:769] Sending connection context
0927 15:16:26.296755 (+ 322us) client_negotiation.cc:240] Negotiation successful
0927 15:16:26.297329 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":347,"threads_started":1}
I20260927 15:16:26.301615 24986 heartbeater.cc:347] Connected to a master server at 127.10.33.126:35165
I20260927 15:16:26.301934 24986 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:26.302649 24986 heartbeater.cc:511] Master 127.10.33.126:35165 requested a full tablet report, sending...
I20260927 15:16:26.304667 24801 ts_manager.cc:194] Registered new tserver with Master: fa90345f69fa494eb8dbcde852d342e8 (127.10.33.65:45281)
I20260927 15:16:26.306234 24801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:59341
W20260927 15:16:26.604377 24990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:26.604700 24990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:26.604823 24990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:26.612192 24990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:26.612340 24990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:26.612488 24990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:16:26.622686 24990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:43741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=46595
--tserver_master_addrs=127.10.33.126:35165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:26.623983 24990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:26.625254 24990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:26.638504 24995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:26.638479 24998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:26.638566 24996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:26.639547 24990 server_base.cc:1034] running on GCE node
I20260927 15:16:26.640166 24990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:26.641204 24990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:26.642421 24990 hybrid_clock.cc:648] HybridClock initialized: now 1790522186642364 us; error 42 us; skew 500 ppm
I20260927 15:16:26.642800 24990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:26 dist-test-slave-f5h6 krb5kdc[24218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522186, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:26.646096 24990 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:16:26.647716 24990 webserver.cc:460] Webserver started at http://127.10.33.66:46595/ using document root <none> and password file <none>
I20260927 15:16:26.648429 24990 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:26.648590 24990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:26.654608 24990 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:16:26.657670 25005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:26.659229 24990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:16:26.659415 24990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "c5568bb7d7144694bb0eb96ea72d21fd"
format_stamp: "Formatted at 2026-09-27 15:16:23 on dist-test-slave-f5h6"
server_key: "3b3c04363c29f595821a1c5e7bd2c0bb"
server_key_iv: "ab1c47ce8e9bc2ae05cb08301861ef18"
server_key_version: "encryptionkey@0"
I20260927 15:16:26.660460 24990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:26.673280 24990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:26.677775 24990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:26.678087 24990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:26.679170 24990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:26.680599 24990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:26.680701 24990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:26.680811 24990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:26.680882 24990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:26.712105 24990 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:43741
I20260927 15:16:26.712184 25117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:43741 every 8 connection(s)
I20260927 15:16:26.713583 24990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:16:26.714516 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 24990
I20260927 15:16:26.715229 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:43765
--local_ip_for_outbound_sockets=127.10.33.67
--tserver_master_addrs=127.10.33.126:35165
--webserver_port=43829
--webserver_interface=127.10.33.67
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:16:26 dist-test-slave-f5h6 krb5kdc[24218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522186, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:16:26.734844 24988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:26.716204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:45783 (local address 127.10.33.126:35165)
0927 15:16:26.716413 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:26.716420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:26.717626 (+ 1206us) server_negotiation.cc:407] Connection header received
0927 15:16:26.720069 (+ 2443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:26.720075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:26.720162 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:26.720263 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:26.722306 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:26.723127 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:26.724754 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:26.725181 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:26.725296 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:26.728961 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:26.728966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:26.728970 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:26.729007 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:26.731788 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:26.732534 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:26.732538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:26.732541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:26.732674 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:26.733084 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:26.733087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:26.733089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:26.733359 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:26.733490 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:26.734253 (+ 763us) server_negotiation.cc:299] Negotiation successful
0927 15:16:26.734485 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:16:26.735927 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:26.716655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.66:45783)
0927 15:16:26.717444 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:26.717489 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:26.719763 (+ 2274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:26.720516 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:26.720529 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:26.721273 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:26.722128 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:26.722154 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:26.723301 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:26.723305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:26.724573 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:26.724586 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:26.725416 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:26.725423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:26.725648 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:26.726385 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:26.726411 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:26.728712 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:26.731938 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:26.731947 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:26.731967 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:26.732407 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:26.732777 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:26.732782 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:26.732785 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:26.732969 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:26.733488 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:26.733496 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:26.733836 (+ 340us) client_negotiation.cc:769] Sending connection context
0927 15:16:26.734129 (+ 293us) client_negotiation.cc:240] Negotiation successful
0927 15:16:26.734689 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":286,"threads_started":1}
I20260927 15:16:26.737440 25118 heartbeater.cc:347] Connected to a master server at 127.10.33.126:35165
I20260927 15:16:26.737753 25118 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:26.738519 25118 heartbeater.cc:511] Master 127.10.33.126:35165 requested a full tablet report, sending...
I20260927 15:16:26.740041 24801 ts_manager.cc:194] Registered new tserver with Master: c5568bb7d7144694bb0eb96ea72d21fd (127.10.33.66:43741)
I20260927 15:16:26.740849 24801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:45783
W20260927 15:16:27.038161 25121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:27.038453 25121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:27.038594 25121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:27.045750 25121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:27.045909 25121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:27.046069 25121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:16:27.055900 25121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:43765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=43829
--tserver_master_addrs=127.10.33.126:35165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:27.057297 25121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:27.058796 25121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:27.071861 25126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:27.071920 25127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:27.072080 25129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:27.072744 25121 server_base.cc:1034] running on GCE node
I20260927 15:16:27.073354 25121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:27.074364 25121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:27.075582 25121 hybrid_clock.cc:648] HybridClock initialized: now 1790522187075519 us; error 57 us; skew 500 ppm
I20260927 15:16:27.075903 25121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:27 dist-test-slave-f5h6 krb5kdc[24218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522187, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:27.078945 25121 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:16:27.080315 25121 webserver.cc:460] Webserver started at http://127.10.33.67:43829/ using document root <none> and password file <none>
I20260927 15:16:27.080950 25121 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:27.081054 25121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:27.086233 25121 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:16:27.088876 25136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:27.090233 25121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:16:27.090368 25121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "713756811bba484baf1c6147e0f9da93"
format_stamp: "Formatted at 2026-09-27 15:16:24 on dist-test-slave-f5h6"
server_key: "3d986d924d31c1b11cbe10bd48372eb6"
server_key_iv: "741048f6c45c3b10e3b6dfd6a99ab352"
server_key_version: "encryptionkey@0"
I20260927 15:16:27.091312 25121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:27.112814 25121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:27.117228 25121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:27.117496 25121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:27.118646 25121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:27.120054 25121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:27.120150 25121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:27.120242 25121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:27.120309 25121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:27.153631 25121 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:43765
I20260927 15:16:27.153759 25248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:43765 every 8 connection(s)
I20260927 15:16:27.155256 25121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:16:27.159690 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 25121
Sep 27 15:16:27 dist-test-slave-f5h6 krb5kdc[24218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522187, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:16:27.175969 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:27.160751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35374 (local address 127.10.33.126:35165)
0927 15:16:27.161297 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:27.161303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:27.161635 (+ 332us) server_negotiation.cc:407] Connection header received
0927 15:16:27.161785 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:27.161791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:27.161922 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:27.162043 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:27.163599 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.164617 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.166347 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.166992 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.167128 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:27.170026 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:27.170031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:27.170034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:27.170067 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:27.172900 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:27.173781 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:27.173785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:27.173787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:27.173980 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:27.174413 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:27.174418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:27.174420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:27.174744 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:27.175014 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:27.175352 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 15:16:27.175580 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":249,"threads_started":1}
I20260927 15:16:27.177484 24988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:27.158189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:48079 (local address 127.10.33.126:35165)
0927 15:16:27.158416 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:27.158421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:27.159865 (+ 1444us) server_negotiation.cc:407] Connection header received
0927 15:16:27.162264 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:27.162269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:27.162329 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:27.162413 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:27.164221 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.165204 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.167315 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.167947 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.168069 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:27.171383 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:27.171388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:27.171391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:27.171432 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:27.174354 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:27.175051 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:27.175056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:27.175060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:27.175165 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:27.175576 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:27.175581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:27.175584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:27.175879 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:27.176012 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:27.176964 (+ 952us) server_negotiation.cc:299] Negotiation successful
0927 15:16:27.177175 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 15:16:27.178694 25251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:27.158711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.67:48079)
0927 15:16:27.159598 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:27.159663 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:27.161931 (+ 2268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:27.162681 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:27.162691 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:27.163252 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:27.164036 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:27.164065 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.165396 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.165400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:27.167110 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:27.167125 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.168180 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.168186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:27.168407 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:27.168982 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:27.169012 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:27.171162 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:27.174467 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:27.174477 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:27.174497 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:27.174929 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:27.175270 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:27.175275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:27.175279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:27.175463 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:27.176023 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:27.176032 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:27.176524 (+ 492us) client_negotiation.cc:769] Sending connection context
0927 15:16:27.176838 (+ 314us) client_negotiation.cc:240] Negotiation successful
0927 15:16:27.177306 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":318,"threads_started":1}
I20260927 15:16:27.180184 25249 heartbeater.cc:347] Connected to a master server at 127.10.33.126:35165
I20260927 15:16:27.180506 25249 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:27.181290 25249 heartbeater.cc:511] Master 127.10.33.126:35165 requested a full tablet report, sending...
I20260927 15:16:27.182981 24801 ts_manager.cc:194] Registered new tserver with Master: 713756811bba484baf1c6147e0f9da93 (127.10.33.67:43765)
I20260927 15:16:27.183804 24801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:48079
I20260927 15:16:27.191625 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:16:27.199259 24988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:27.192698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35380 (local address 127.10.33.126:35165)
0927 15:16:27.192871 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:27.192875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:27.193341 (+ 466us) server_negotiation.cc:407] Connection header received
0927 15:16:27.193481 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:27.193485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:27.193724 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:27.193940 (+ 216us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:16:27.194526 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.195333 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.196974 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.197444 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.197550 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:27.198201 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:16:27.198756 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:16:27.198872 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:27.198971 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 15:16:27.199048 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:16:27.200314 24801 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:16:27.207605 24801 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:16:27.230808 25261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:27.220653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.10.33.67:43765)
0927 15:16:27.221681 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:27.221689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:27.221723 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:16:27.221911 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:27.221919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:27.222190 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:27.223671 (+ 1481us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:27.224020 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.225321 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.228202 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.229435 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.229568 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:27.229834 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:27.230318 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 15:16:27.230485 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":858,"thread_start_us":702,"threads_started":1}
I20260927 15:16:27.230836 25260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:27.220196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:43765 (local address 127.0.0.1:35358)
0927 15:16:27.221415 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:27.221457 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:27.221665 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:27.222454 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:27.222463 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:27.222499 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:27.223026 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:27.223040 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.225490 (+ 2450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.225496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:27.227960 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:27.227980 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.229643 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.229650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:27.229835 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:27.229947 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 15:16:27.230161 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 15:16:27.230395 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"mutex_wait_us":6,"thread_start_us":1847,"threads_started":1}
I20260927 15:16:27.231468 25263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:27.222453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:43741 (local address 127.0.0.1:37934)
0927 15:16:27.223027 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:27.223043 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:27.223149 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:27.224177 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:27.224182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:27.224206 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:27.224554 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:27.224578 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.226408 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.226413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:27.228616 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:27.228631 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.230415 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.230421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:27.230610 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:27.230645 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 15:16:27.230714 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 15:16:27.230854 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":803,"threads_started":1}
I20260927 15:16:27.232018 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:27.220599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:45281 (local address 127.0.0.1:49886)
0927 15:16:27.222297 (+ 1698us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:27.222314 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:27.222413 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:27.225346 (+ 2933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:27.225351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:27.225376 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:27.225882 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:27.225895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.227393 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.227398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:27.229317 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:27.229331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.231224 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.231232 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:27.231398 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:27.231429 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 15:16:27.231495 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 15:16:27.231632 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1566,"mutex_wait_us":26,"thread_start_us":731,"threads_started":1}
I20260927 15:16:27.232553 25264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:27.223098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37934 (local address 127.10.33.66:43741)
0927 15:16:27.223592 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:27.223600 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:27.223632 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:16:27.223724 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:27.223732 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:27.223962 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:27.224754 (+ 792us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:27.225025 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.226252 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.228776 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.230187 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.231722 (+ 1535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:27.231943 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:27.232014 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 15:16:27.232240 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":269,"threads_started":1}
I20260927 15:16:27.232904 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:27.221162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49886 (local address 127.10.33.65:45281)
0927 15:16:27.224761 (+ 3599us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:27.224769 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:27.224795 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:16:27.224895 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:27.224902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:27.225140 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:27.225335 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:27.226078 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.227231 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.229515 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.230975 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.232238 (+ 1263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:27.232422 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:27.232485 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 15:16:27.232638 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3451,"thread_start_us":3362,"threads_started":1}
I20260927 15:16:27.235093 25184 tablet_service.cc:1467] Processing CreateTablet for tablet cf611118a3ca4c12a51959b9f22cdbde (DEFAULT_TABLE table=test-table [id=89ee2bba2f2349e1a3c99b04ecc8f6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:27.236037 25053 tablet_service.cc:1467] Processing CreateTablet for tablet cf611118a3ca4c12a51959b9f22cdbde (DEFAULT_TABLE table=test-table [id=89ee2bba2f2349e1a3c99b04ecc8f6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:27.236821 25184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf611118a3ca4c12a51959b9f22cdbde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:27.237932 25053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf611118a3ca4c12a51959b9f22cdbde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:27.237772 24921 tablet_service.cc:1467] Processing CreateTablet for tablet cf611118a3ca4c12a51959b9f22cdbde (DEFAULT_TABLE table=test-table [id=89ee2bba2f2349e1a3c99b04ecc8f6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:27.238876 24921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf611118a3ca4c12a51959b9f22cdbde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:27.250229 25265 tablet_bootstrap.cc:492] T cf611118a3ca4c12a51959b9f22cdbde P fa90345f69fa494eb8dbcde852d342e8: Bootstrap starting.
I20260927 15:16:27.252317 25266 tablet_bootstrap.cc:492] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93: Bootstrap starting.
I20260927 15:16:27.253618 25267 tablet_bootstrap.cc:492] T cf611118a3ca4c12a51959b9f22cdbde P c5568bb7d7144694bb0eb96ea72d21fd: Bootstrap starting.
I20260927 15:16:27.254616 25265 tablet_bootstrap.cc:654] T cf611118a3ca4c12a51959b9f22cdbde P fa90345f69fa494eb8dbcde852d342e8: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:27.255832 25266 tablet_bootstrap.cc:654] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:27.255898 25265 log.cc:824] T cf611118a3ca4c12a51959b9f22cdbde P fa90345f69fa494eb8dbcde852d342e8: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:27.256335 25267 tablet_bootstrap.cc:654] T cf611118a3ca4c12a51959b9f22cdbde P c5568bb7d7144694bb0eb96ea72d21fd: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:27.257040 25266 log.cc:824] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:27.257395 25267 log.cc:824] T cf611118a3ca4c12a51959b9f22cdbde P c5568bb7d7144694bb0eb96ea72d21fd: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:27.259683 25266 tablet_bootstrap.cc:492] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93: No bootstrap required, opened a new log
I20260927 15:16:27.259819 25265 tablet_bootstrap.cc:492] T cf611118a3ca4c12a51959b9f22cdbde P fa90345f69fa494eb8dbcde852d342e8: No bootstrap required, opened a new log
I20260927 15:16:27.259974 25266 ts_tablet_manager.cc:1397] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260927 15:16:27.260089 25265 ts_tablet_manager.cc:1397] T cf611118a3ca4c12a51959b9f22cdbde P fa90345f69fa494eb8dbcde852d342e8: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260927 15:16:27.263279 25267 tablet_bootstrap.cc:492] T cf611118a3ca4c12a51959b9f22cdbde P c5568bb7d7144694bb0eb96ea72d21fd: No bootstrap required, opened a new log
I20260927 15:16:27.263536 25267 ts_tablet_manager.cc:1397] T cf611118a3ca4c12a51959b9f22cdbde P c5568bb7d7144694bb0eb96ea72d21fd: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260927 15:16:27.268198 25266 raft_consensus.cc:348] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713756811bba484baf1c6147e0f9da93" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43765 } } peers { permanent_uuid: "fa90345f69fa494eb8dbcde852d342e8" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45281 } } peers { permanent_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 43741 } }
I20260927 15:16:27.268572 25266 raft_consensus.cc:374] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:27.268687 25266 raft_consensus.cc:729] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 713756811bba484baf1c6147e0f9da93, State: Initialized, Role: FOLLOWER
I20260927 15:16:27.269268 25266 consensus_queue.cc:260] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713756811bba484baf1c6147e0f9da93" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43765 } } peers { permanent_uuid: "fa90345f69fa494eb8dbcde852d342e8" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45281 } } peers { permanent_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 43741 } }
I20260927 15:16:27.270117 25265 raft_consensus.cc:348] T cf611118a3ca4c12a51959b9f22cdbde P fa90345f69fa494eb8dbcde852d342e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713756811bba484baf1c6147e0f9da93" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43765 } } peers { permanent_uuid: "fa90345f69fa494eb8dbcde852d342e8" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45281 } } peers { permanent_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 43741 } }
I20260927 15:16:27.270602 25265 raft_consensus.cc:374] T cf611118a3ca4c12a51959b9f22cdbde P fa90345f69fa494eb8dbcde852d342e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:27.270728 25265 raft_consensus.cc:729] T cf611118a3ca4c12a51959b9f22cdbde P fa90345f69fa494eb8dbcde852d342e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa90345f69fa494eb8dbcde852d342e8, State: Initialized, Role: FOLLOWER
I20260927 15:16:27.271026 25266 ts_tablet_manager.cc:1428] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260927 15:16:27.271554 25249 heartbeater.cc:503] Master 127.10.33.126:35165 was elected leader, sending a full tablet report...
I20260927 15:16:27.271425 25265 consensus_queue.cc:260] T cf611118a3ca4c12a51959b9f22cdbde P fa90345f69fa494eb8dbcde852d342e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713756811bba484baf1c6147e0f9da93" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43765 } } peers { permanent_uuid: "fa90345f69fa494eb8dbcde852d342e8" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45281 } } peers { permanent_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 43741 } }
I20260927 15:16:27.272454 24986 heartbeater.cc:503] Master 127.10.33.126:35165 was elected leader, sending a full tablet report...
I20260927 15:16:27.273106 25265 ts_tablet_manager.cc:1428] T cf611118a3ca4c12a51959b9f22cdbde P fa90345f69fa494eb8dbcde852d342e8: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260927 15:16:27.274914 25267 raft_consensus.cc:348] T cf611118a3ca4c12a51959b9f22cdbde P c5568bb7d7144694bb0eb96ea72d21fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713756811bba484baf1c6147e0f9da93" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43765 } } peers { permanent_uuid: "fa90345f69fa494eb8dbcde852d342e8" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45281 } } peers { permanent_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 43741 } }
I20260927 15:16:27.275357 25267 raft_consensus.cc:374] T cf611118a3ca4c12a51959b9f22cdbde P c5568bb7d7144694bb0eb96ea72d21fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:27.275458 25267 raft_consensus.cc:729] T cf611118a3ca4c12a51959b9f22cdbde P c5568bb7d7144694bb0eb96ea72d21fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5568bb7d7144694bb0eb96ea72d21fd, State: Initialized, Role: FOLLOWER
I20260927 15:16:27.276093 25267 consensus_queue.cc:260] T cf611118a3ca4c12a51959b9f22cdbde P c5568bb7d7144694bb0eb96ea72d21fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713756811bba484baf1c6147e0f9da93" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43765 } } peers { permanent_uuid: "fa90345f69fa494eb8dbcde852d342e8" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45281 } } peers { permanent_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 43741 } }
W20260927 15:16:27.276875 24987 tablet.cc:2369] T cf611118a3ca4c12a51959b9f22cdbde P fa90345f69fa494eb8dbcde852d342e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:16:27.280216 25118 heartbeater.cc:503] Master 127.10.33.126:35165 was elected leader, sending a full tablet report...
I20260927 15:16:27.280853 25267 ts_tablet_manager.cc:1428] T cf611118a3ca4c12a51959b9f22cdbde P c5568bb7d7144694bb0eb96ea72d21fd: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
W20260927 15:16:27.407058 25250 tablet.cc:2369] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:16:27.440208 25271 raft_consensus.cc:482] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:16:27.440603 25271 raft_consensus.cc:504] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713756811bba484baf1c6147e0f9da93" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43765 } } peers { permanent_uuid: "fa90345f69fa494eb8dbcde852d342e8" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45281 } } peers { permanent_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 43741 } }
I20260927 15:16:27.442375 25271 leader_election.cc:288] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa90345f69fa494eb8dbcde852d342e8 (127.10.33.65:45281), c5568bb7d7144694bb0eb96ea72d21fd (127.10.33.66:43741)
I20260927 15:16:27.447757 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:27.442736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:59047 (local address 127.10.33.65:45281)
0927 15:16:27.442939 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:27.442945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:27.443054 (+ 109us) server_negotiation.cc:407] Connection header received
0927 15:16:27.443285 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:27.443289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:27.443347 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:27.443418 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:27.443819 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.444527 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.446125 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.446912 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.447054 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:27.447128 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:27.447388 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 15:16:27.447513 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:16:27.447772 25251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:27.442623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:45281 (local address 127.10.33.67:59047)
0927 15:16:27.442960 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:27.442978 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:27.443123 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:27.443453 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:27.443457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:27.443488 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:27.443657 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:27.443692 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.444647 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.444653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:27.445983 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:27.445995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.447054 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.447060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:27.447229 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:27.447265 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:16:27.447344 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 15:16:27.447475 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210}
I20260927 15:16:27.448401 25264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:27.443003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:35885 (local address 127.10.33.66:43741)
0927 15:16:27.443207 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:27.443213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:27.443595 (+ 382us) server_negotiation.cc:407] Connection header received
0927 15:16:27.443758 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:27.443761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:27.443819 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:27.443891 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:27.444378 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.445042 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.446713 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.447538 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.447684 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:27.447765 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:27.448058 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 15:16:27.448199 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:16:27.448419 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:27.442802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:43741 (local address 127.10.33.67:35885)
0927 15:16:27.443494 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:27.443507 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:27.443632 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:27.443918 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:27.443922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:27.443944 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:27.444239 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:27.444248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.445185 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.445189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:27.446552 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:27.446564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.447729 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.447736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:27.447900 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:27.447936 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:16:27.448023 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 15:16:27.448144 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":402,"threads_started":1}
I20260927 15:16:27.448881 24941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf611118a3ca4c12a51959b9f22cdbde" candidate_uuid: "713756811bba484baf1c6147e0f9da93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa90345f69fa494eb8dbcde852d342e8" is_pre_election: true
I20260927 15:16:27.449486 24941 raft_consensus.cc:2454] T cf611118a3ca4c12a51959b9f22cdbde P fa90345f69fa494eb8dbcde852d342e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 713756811bba484baf1c6147e0f9da93 in term 0.
I20260927 15:16:27.449474 25073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf611118a3ca4c12a51959b9f22cdbde" candidate_uuid: "713756811bba484baf1c6147e0f9da93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" is_pre_election: true
I20260927 15:16:27.449990 25073 raft_consensus.cc:2454] T cf611118a3ca4c12a51959b9f22cdbde P c5568bb7d7144694bb0eb96ea72d21fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 713756811bba484baf1c6147e0f9da93 in term 0.
I20260927 15:16:27.450284 25139 leader_election.cc:302] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 713756811bba484baf1c6147e0f9da93, fa90345f69fa494eb8dbcde852d342e8; no voters:
I20260927 15:16:27.450801 25271 raft_consensus.cc:2781] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:16:27.450932 25271 raft_consensus.cc:482] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:16:27.451030 25271 raft_consensus.cc:3037] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:27.453138 25271 raft_consensus.cc:504] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713756811bba484baf1c6147e0f9da93" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43765 } } peers { permanent_uuid: "fa90345f69fa494eb8dbcde852d342e8" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45281 } } peers { permanent_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 43741 } }
I20260927 15:16:27.453715 25271 leader_election.cc:288] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93 [CANDIDATE]: Term 1 election: Requested vote from peers fa90345f69fa494eb8dbcde852d342e8 (127.10.33.65:45281), c5568bb7d7144694bb0eb96ea72d21fd (127.10.33.66:43741)
I20260927 15:16:27.454173 24941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf611118a3ca4c12a51959b9f22cdbde" candidate_uuid: "713756811bba484baf1c6147e0f9da93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa90345f69fa494eb8dbcde852d342e8"
I20260927 15:16:27.454209 25073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf611118a3ca4c12a51959b9f22cdbde" candidate_uuid: "713756811bba484baf1c6147e0f9da93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5568bb7d7144694bb0eb96ea72d21fd"
I20260927 15:16:27.454387 25073 raft_consensus.cc:3037] T cf611118a3ca4c12a51959b9f22cdbde P c5568bb7d7144694bb0eb96ea72d21fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:27.454386 24941 raft_consensus.cc:3037] T cf611118a3ca4c12a51959b9f22cdbde P fa90345f69fa494eb8dbcde852d342e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:27.457567 24941 raft_consensus.cc:2454] T cf611118a3ca4c12a51959b9f22cdbde P fa90345f69fa494eb8dbcde852d342e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713756811bba484baf1c6147e0f9da93 in term 1.
I20260927 15:16:27.457567 25073 raft_consensus.cc:2454] T cf611118a3ca4c12a51959b9f22cdbde P c5568bb7d7144694bb0eb96ea72d21fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713756811bba484baf1c6147e0f9da93 in term 1.
I20260927 15:16:27.458047 25137 leader_election.cc:302] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 713756811bba484baf1c6147e0f9da93, c5568bb7d7144694bb0eb96ea72d21fd; no voters:
I20260927 15:16:27.458461 25271 raft_consensus.cc:2781] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:27.459002 25271 raft_consensus.cc:686] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93 [term 1 LEADER]: Becoming Leader. State: Replica: 713756811bba484baf1c6147e0f9da93, State: Running, Role: LEADER
I20260927 15:16:27.459549 25271 consensus_queue.cc:237] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713756811bba484baf1c6147e0f9da93" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43765 } } peers { permanent_uuid: "fa90345f69fa494eb8dbcde852d342e8" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45281 } } peers { permanent_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 43741 } }
I20260927 15:16:27.464303 24800 catalog_manager.cc:5495] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93 reported cstate change: term changed from 0 to 1, leader changed from <none> to 713756811bba484baf1c6147e0f9da93 (127.10.33.67). New cstate: current_term: 1 leader_uuid: "713756811bba484baf1c6147e0f9da93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713756811bba484baf1c6147e0f9da93" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 43765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa90345f69fa494eb8dbcde852d342e8" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 43741 } health_report { overall_health: UNKNOWN } } }
W20260927 15:16:27.465512 25119 tablet.cc:2369] T cf611118a3ca4c12a51959b9f22cdbde P c5568bb7d7144694bb0eb96ea72d21fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:16:27.527807 25261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:27.522472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.10.33.67:43765)
0927 15:16:27.522702 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:27.522707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:27.522727 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:16:27.522930 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:27.522934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:27.523018 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:27.523144 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:16:27.523513 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.524216 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.525598 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.526045 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.526156 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:27.526675 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:16:27.527205 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:16:27.527351 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:27.527448 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 15:16:27.527533 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:16:27.536087 24941 raft_consensus.cc:1261] T cf611118a3ca4c12a51959b9f22cdbde P fa90345f69fa494eb8dbcde852d342e8 [term 1 FOLLOWER]: Refusing update from remote peer 713756811bba484baf1c6147e0f9da93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:27.536085 25073 raft_consensus.cc:1261] T cf611118a3ca4c12a51959b9f22cdbde P c5568bb7d7144694bb0eb96ea72d21fd [term 1 FOLLOWER]: Refusing update from remote peer 713756811bba484baf1c6147e0f9da93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:27.537094 25275 consensus_queue.cc:1035] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa90345f69fa494eb8dbcde852d342e8" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:27.537484 25271 consensus_queue.cc:1035] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5568bb7d7144694bb0eb96ea72d21fd" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 43741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:27.548658 25278 mvcc.cc:205] Tried to move back new op lower bound from 7333978880137453568 to 7333978879837999104. Current Snapshot: MvccSnapshot[applied={T|T < 7333978880137453568}]
I20260927 15:16:27.549990 25280 mvcc.cc:205] Tried to move back new op lower bound from 7333978880137453568 to 7333978879837999104. Current Snapshot: MvccSnapshot[applied={T|T < 7333978880137453568}]
I20260927 15:16:27.554066 25279 mvcc.cc:205] Tried to move back new op lower bound from 7333978880137453568 to 7333978879837999104. Current Snapshot: MvccSnapshot[applied={T|T < 7333978880137453568}]
I20260927 15:16:27.562840 24801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:27.563133 24801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:27.566748 24801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295: Sending DeleteTablet for 3 replicas of tablet cf611118a3ca4c12a51959b9f22cdbde
I20260927 15:16:27.568045 24921 tablet_service.cc:1514] Processing DeleteTablet for tablet cf611118a3ca4c12a51959b9f22cdbde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:27 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:49886
I20260927 15:16:27.568274 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 24838
I20260927 15:16:27.568800 25184 tablet_service.cc:1514] Processing DeleteTablet for tablet cf611118a3ca4c12a51959b9f22cdbde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:27 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:35358
I20260927 15:16:27.568946 25287 tablet_replica.cc:330] T cf611118a3ca4c12a51959b9f22cdbde P fa90345f69fa494eb8dbcde852d342e8: stopping tablet replica
I20260927 15:16:27.569629 25287 raft_consensus.cc:2229] T cf611118a3ca4c12a51959b9f22cdbde P fa90345f69fa494eb8dbcde852d342e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:16:27.569893 25053 tablet_service.cc:1514] Processing DeleteTablet for tablet cf611118a3ca4c12a51959b9f22cdbde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:27 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:37934
I20260927 15:16:27.570241 25287 raft_consensus.cc:2258] T cf611118a3ca4c12a51959b9f22cdbde P fa90345f69fa494eb8dbcde852d342e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:27.572481 25287 ts_tablet_manager.cc:1905] T cf611118a3ca4c12a51959b9f22cdbde P fa90345f69fa494eb8dbcde852d342e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:27.580742 25287 ts_tablet_manager.cc:1918] T cf611118a3ca4c12a51959b9f22cdbde P fa90345f69fa494eb8dbcde852d342e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:16:27.581012 25287 log.cc:1196] T cf611118a3ca4c12a51959b9f22cdbde P fa90345f69fa494eb8dbcde852d342e8: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/cf611118a3ca4c12a51959b9f22cdbde
I20260927 15:16:27.581652 25287 ts_tablet_manager.cc:1939] T cf611118a3ca4c12a51959b9f22cdbde P fa90345f69fa494eb8dbcde852d342e8: Deleting consensus metadata
I20260927 15:16:27.584463 24787 catalog_manager.cc:4841] TS fa90345f69fa494eb8dbcde852d342e8 (127.10.33.65:45281): tablet cf611118a3ca4c12a51959b9f22cdbde (table test-table [id=89ee2bba2f2349e1a3c99b04ecc8f6b7]) successfully deleted
I20260927 15:16:27.592176 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:27.569219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49900 (local address 127.10.33.65:45281)
0927 15:16:27.569559 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:27.569564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:27.569579 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:16:27.569638 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:27.569643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:27.569758 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:27.569887 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:27.571813 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.573001 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.575163 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.575860 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.575987 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:27.579925 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:27.579948 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:27.579972 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:27.580025 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:27.584638 (+ 4613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:27.585809 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:27.585821 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:27.585826 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:27.586050 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:27.587636 (+ 1586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:27.587642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:27.587645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:27.588340 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:27.588759 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:27.591462 (+ 2703us) server_negotiation.cc:299] Negotiation successful
0927 15:16:27.591752 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245}
I20260927 15:16:27.593437 25288 tablet_replica.cc:330] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93: stopping tablet replica
I20260927 15:16:27.593475 24981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:27.594074 25288 raft_consensus.cc:2229] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:16:27.594980 25288 raft_consensus.cc:2258] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:27.596243 25289 tablet_replica.cc:330] T cf611118a3ca4c12a51959b9f22cdbde P c5568bb7d7144694bb0eb96ea72d21fd: stopping tablet replica
I20260927 15:16:27.596758 25289 raft_consensus.cc:2229] T cf611118a3ca4c12a51959b9f22cdbde P c5568bb7d7144694bb0eb96ea72d21fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:16:27.597347 25289 raft_consensus.cc:2258] T cf611118a3ca4c12a51959b9f22cdbde P c5568bb7d7144694bb0eb96ea72d21fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:27.597474 25288 ts_tablet_manager.cc:1905] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:27.600140 25289 ts_tablet_manager.cc:1905] T cf611118a3ca4c12a51959b9f22cdbde P c5568bb7d7144694bb0eb96ea72d21fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:27.605434 25288 ts_tablet_manager.cc:1918] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:16:27.605757 25288 log.cc:1196] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/cf611118a3ca4c12a51959b9f22cdbde
I20260927 15:16:27.606467 25288 ts_tablet_manager.cc:1939] T cf611118a3ca4c12a51959b9f22cdbde P 713756811bba484baf1c6147e0f9da93: Deleting consensus metadata
I20260927 15:16:27.608659 24785 catalog_manager.cc:4841] TS 713756811bba484baf1c6147e0f9da93 (127.10.33.67:43765): tablet cf611118a3ca4c12a51959b9f22cdbde (table test-table [id=89ee2bba2f2349e1a3c99b04ecc8f6b7]) successfully deleted
I20260927 15:16:27.608728 25289 ts_tablet_manager.cc:1918] T cf611118a3ca4c12a51959b9f22cdbde P c5568bb7d7144694bb0eb96ea72d21fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:16:27.608922 25289 log.cc:1196] T cf611118a3ca4c12a51959b9f22cdbde P c5568bb7d7144694bb0eb96ea72d21fd: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/cf611118a3ca4c12a51959b9f22cdbde
I20260927 15:16:27.609549 25289 ts_tablet_manager.cc:1939] T cf611118a3ca4c12a51959b9f22cdbde P c5568bb7d7144694bb0eb96ea72d21fd: Deleting consensus metadata
I20260927 15:16:27.611593 24785 catalog_manager.cc:4841] TS c5568bb7d7144694bb0eb96ea72d21fd (127.10.33.66:43741): tablet cf611118a3ca4c12a51959b9f22cdbde (table test-table [id=89ee2bba2f2349e1a3c99b04ecc8f6b7]) successfully deleted
I20260927 15:16:27.702087 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 24838
I20260927 15:16:27.724491 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 24990
I20260927 15:16:27.745294 25264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:27.725337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37938 (local address 127.10.33.66:43741)
0927 15:16:27.725967 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:27.725974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:27.725992 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:16:27.726047 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:27.726052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:27.726122 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:27.726216 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:27.728007 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.729184 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.731377 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.731946 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.732055 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:27.739413 (+ 7358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:27.739430 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:27.739446 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:27.739495 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:27.742460 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:27.743360 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:27.743368 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:27.743371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:27.743504 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:27.743884 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:27.743887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:27.743890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:27.744270 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:27.744485 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:27.744854 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 15:16:27.745020 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474}
I20260927 15:16:27.745955 25113 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:27.847608 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 24990
I20260927 15:16:27.869673 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 25121
I20260927 15:16:27.886127 25261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:27.870457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35366 (local address 127.10.33.67:43765)
0927 15:16:27.870639 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:27.870644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:27.870751 (+ 107us) server_negotiation.cc:407] Connection header received
0927 15:16:27.870937 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:27.870941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:27.871024 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:27.871101 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:27.873004 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.873914 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.875885 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:27.876362 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:27.876469 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:27.880005 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:27.880019 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:27.880035 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:27.880088 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:27.883141 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:27.884019 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:27.884027 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:27.884029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:27.884211 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:27.884587 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:27.884590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:27.884592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:27.884938 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:27.885153 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:27.885511 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 15:16:27.885722 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 15:16:27.886825 25244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:28.011586 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 25121
I20260927 15:16:28.028486 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 24768
I20260927 15:16:28.029650 24831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:28.143270 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 24768
I20260927 15:16:28.161751 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:35165
--webserver_interface=127.10.33.126
--webserver_port=41623
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:35165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:28.492077 25294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:28.492389 25294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:28.492514 25294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:28.499914 25294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:28.500029 25294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:28.500075 25294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:28.500130 25294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:28.500173 25294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:16:28.510893 25294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:35165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:35165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=41623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:28.512253 25294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:28.513613 25294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:28.525272 25299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:28.525247 25300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:28.525147 25302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:28.526144 25294 server_base.cc:1034] running on GCE node
I20260927 15:16:28.526903 25294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:28.528314 25294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:28.529547 25294 hybrid_clock.cc:648] HybridClock initialized: now 1790522188529497 us; error 31 us; skew 500 ppm
I20260927 15:16:28.529925 25294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:28 dist-test-slave-f5h6 krb5kdc[24218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522188, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:28.533066 25294 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:16:28.534796 25294 webserver.cc:460] Webserver started at http://127.10.33.126:41623/ using document root <none> and password file <none>
I20260927 15:16:28.535526 25294 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:28.535652 25294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:28.541307 25294 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:16:28.544492 25309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:28.545979 25294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:16:28.546176 25294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "e5a5cd3ea7144cac9eb3e472f7f51295"
format_stamp: "Formatted at 2026-09-27 15:16:22 on dist-test-slave-f5h6"
server_key: "2e5ed7f594e28d5dcc658b6c680dac79"
server_key_iv: "e8c8e4fa99ecbd10fd1052170a529cb3"
server_key_version: "encryptionkey@0"
I20260927 15:16:28.547192 25294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:28.572445 25294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:28.576869 25294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:28.577131 25294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:28.595460 25294 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:35165
I20260927 15:16:28.595472 25360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:35165 every 8 connection(s)
I20260927 15:16:28.596972 25294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:16:28.603997 25361 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:16:28.604992 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 25294
I20260927 15:16:28.605947 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:45281
--local_ip_for_outbound_sockets=127.10.33.65
--tserver_master_addrs=127.10.33.126:35165
--webserver_port=39529
--webserver_interface=127.10.33.65
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:16:28.607741 25361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295: Bootstrap starting.
I20260927 15:16:28.633761 25361 log.cc:824] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:28.655313 25361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:16:28.656013 25361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295: Bootstrap complete.
I20260927 15:16:28.669807 25361 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35165 } }
I20260927 15:16:28.671288 25361 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5a5cd3ea7144cac9eb3e472f7f51295, State: Initialized, Role: FOLLOWER
I20260927 15:16:28.672178 25361 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35165 } }
I20260927 15:16:28.672492 25361 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:16:28.672634 25361 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:16:28.672848 25361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 2 FOLLOWER]: Advancing to term 3
I20260927 15:16:28.678733 25361 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35165 } }
I20260927 15:16:28.679342 25361 leader_election.cc:302] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5a5cd3ea7144cac9eb3e472f7f51295; no voters:
I20260927 15:16:28.680028 25361 leader_election.cc:288] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 15:16:28.680161 25365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 15:16:28.681273 25365 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [term 3 LEADER]: Becoming Leader. State: Replica: e5a5cd3ea7144cac9eb3e472f7f51295, State: Running, Role: LEADER
I20260927 15:16:28.681787 25365 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35165 } }
I20260927 15:16:28.682220 25361 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:28.684195 25366 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35165 } } }
I20260927 15:16:28.684201 25367 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5a5cd3ea7144cac9eb3e472f7f51295. Latest consensus state: current_term: 3 leader_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a5cd3ea7144cac9eb3e472f7f51295" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35165 } } }
I20260927 15:16:28.684602 25367 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:28.684618 25366 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5a5cd3ea7144cac9eb3e472f7f51295 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:28.687170 25370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:16:28.695617 25370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:16:28.697223 25370 catalog_manager.cc:1259] Loaded cluster ID: ed96f4cbbb8840d0b2166907e5a90c55
I20260927 15:16:28.697409 25370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 15:16:28.700028 25370 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4d5a19e9ad google::LogMessage::Fail() at ??:0
@ 0x7f4d5a1a188a google::LogMessage::SendToLog() at ??:0
@ 0x7f4d5a19e4de google::LogMessage::Flush() at ??:0
@ 0x7f4d5a19fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4d6fbd5512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f4d6fbd2976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4d6fc5a610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
I20260927 15:16:28.728279 25383 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f4d6fc5a439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4d5cdea345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4d5cded91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f4d5cded799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4d5cdc8eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4d600a26db start_thread at ??:0
@ 0x7f4d5903861f clone at ??:0
W20260927 15:16:29.014365 25363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:29.014672 25363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:29.014797 25363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:29.022368 25363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:29.022485 25363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:29.022627 25363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:16:29.032799 25363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:45281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=39529
--tserver_master_addrs=127.10.33.126:35165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:29.034217 25363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:29.035583 25363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:29.048861 25402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:29.048935 25403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:29.048907 25405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:29.049965 25363 server_base.cc:1034] running on GCE node
I20260927 15:16:29.050624 25363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:29.051630 25363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:29.052922 25363 hybrid_clock.cc:648] HybridClock initialized: now 1790522189052858 us; error 62 us; skew 500 ppm
I20260927 15:16:29.053251 25363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:29 dist-test-slave-f5h6 krb5kdc[24218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522189, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:29.056437 25363 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:16:29.058099 25363 webserver.cc:460] Webserver started at http://127.10.33.65:39529/ using document root <none> and password file <none>
I20260927 15:16:29.058833 25363 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:29.058964 25363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:29.064965 25363 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 15:16:29.068023 25412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:29.069556 25363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:16:29.069783 25363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "fa90345f69fa494eb8dbcde852d342e8"
format_stamp: "Formatted at 2026-09-27 15:16:23 on dist-test-slave-f5h6"
server_key: "88fa5f86303d37147141adbad5037929"
server_key_iv: "99e396676dafa35cc9c2d001f343615a"
server_key_version: "encryptionkey@0"
I20260927 15:16:29.070888 25363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:29.089257 25363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:29.093786 25363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:29.094080 25363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:29.095129 25363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:29.096498 25363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:29.096612 25363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:29.096720 25363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:29.096794 25363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:29.127965 25363 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:45281
I20260927 15:16:29.128041 25524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:45281 every 8 connection(s)
I20260927 15:16:29.129395 25363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:16:29.134181 25527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:29.132344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.65:45759)
0927 15:16:29.133055 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:29.133280 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":282,"threads_started":1}
I20260927 15:16:29.134192 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 25363
W20260927 15:16:29.134871 25525 heartbeater.cc:644] Failed to heartbeat to 127.10.33.126:35165 (0 consecutive failures): Network error: Failed to ping master at 127.10.33.126:35165: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
I20260927 15:16:29.134917 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:43741
--local_ip_for_outbound_sockets=127.10.33.66
--tserver_master_addrs=127.10.33.126:35165
--webserver_port=46595
--webserver_interface=127.10.33.66
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:16:29.135694 25527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:29.135381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.65:60283)
0927 15:16:29.135519 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:29.135587 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 15:16:29.136569 25527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:29.136178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.65:56651)
0927 15:16:29.136363 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:29.136465 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 15:16:29.136794 25525 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:16:29.457924 25528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:29.458207 25528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:29.458330 25528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:29.465678 25528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:29.465811 25528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:29.465986 25528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:16:29.476104 25528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:43741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=46595
--tserver_master_addrs=127.10.33.126:35165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:29.477444 25528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:29.478821 25528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:29.491940 25533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:29.491869 25536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:29.491899 25534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:29.492436 25528 server_base.cc:1034] running on GCE node
I20260927 15:16:29.493168 25528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:29.494220 25528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:29.495461 25528 hybrid_clock.cc:648] HybridClock initialized: now 1790522189495395 us; error 49 us; skew 500 ppm
I20260927 15:16:29.495812 25528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:29 dist-test-slave-f5h6 krb5kdc[24218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522189, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:29.498932 25528 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:16:29.500504 25528 webserver.cc:460] Webserver started at http://127.10.33.66:46595/ using document root <none> and password file <none>
I20260927 15:16:29.501294 25528 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:29.501415 25528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:29.507009 25528 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:16:29.510149 25543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:29.511662 25528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:16:29.511824 25528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "c5568bb7d7144694bb0eb96ea72d21fd"
format_stamp: "Formatted at 2026-09-27 15:16:23 on dist-test-slave-f5h6"
server_key: "3b3c04363c29f595821a1c5e7bd2c0bb"
server_key_iv: "ab1c47ce8e9bc2ae05cb08301861ef18"
server_key_version: "encryptionkey@0"
I20260927 15:16:29.512835 25528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:29.525919 25528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:29.530181 25528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:29.530433 25528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:29.531365 25528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:29.532671 25528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:29.532768 25528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:29.532874 25528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:29.532943 25528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:29.563532 25528 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:43741
I20260927 15:16:29.563585 25655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:43741 every 8 connection(s)
I20260927 15:16:29.565016 25528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:16:29.568015 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 25528
I20260927 15:16:29.568775 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:43765
--local_ip_for_outbound_sockets=127.10.33.67
--tserver_master_addrs=127.10.33.126:35165
--webserver_port=43829
--webserver_interface=127.10.33.67
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:16:29.569993 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:29.568060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.66:45123)
0927 15:16:29.568816 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:29.569066 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":287,"threads_started":1}
W20260927 15:16:29.570701 25656 heartbeater.cc:644] Failed to heartbeat to 127.10.33.126:35165 (0 consecutive failures): Network error: Failed to ping master at 127.10.33.126:35165: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
I20260927 15:16:29.571585 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:29.571238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.66:53635)
0927 15:16:29.571399 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:29.571471 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 15:16:29.572444 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:29.572114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.66:32915)
0927 15:16:29.572257 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:29.572330 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 15:16:29.572683 25656 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:16:29.890456 25659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:29.890751 25659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:29.890865 25659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:29.898418 25659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:29.898552 25659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:29.898696 25659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:16:29.909313 25659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:43765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=43829
--tserver_master_addrs=127.10.33.126:35165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:29.910722 25659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:29.912287 25659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:29.926244 25665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:29.926331 25664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:29.926265 25667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:29.927017 25659 server_base.cc:1034] running on GCE node
I20260927 15:16:29.927639 25659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:29.928632 25659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:29.929876 25659 hybrid_clock.cc:648] HybridClock initialized: now 1790522189929794 us; error 53 us; skew 500 ppm
I20260927 15:16:29.930184 25659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:29 dist-test-slave-f5h6 krb5kdc[24218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522189, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:29.933209 25659 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:16:29.934715 25659 webserver.cc:460] Webserver started at http://127.10.33.67:43829/ using document root <none> and password file <none>
I20260927 15:16:29.935364 25659 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:29.935462 25659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:29.940834 25659 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:16:29.943842 25674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:29.945230 25659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:16:29.945410 25659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "713756811bba484baf1c6147e0f9da93"
format_stamp: "Formatted at 2026-09-27 15:16:24 on dist-test-slave-f5h6"
server_key: "3d986d924d31c1b11cbe10bd48372eb6"
server_key_iv: "741048f6c45c3b10e3b6dfd6a99ab352"
server_key_version: "encryptionkey@0"
I20260927 15:16:29.946488 25659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:29.981371 25659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:29.985703 25659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:29.985961 25659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:29.986928 25659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:29.988265 25659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:29.988371 25659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:29.988481 25659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:29.988564 25659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:30.023840 25659 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:43765
I20260927 15:16:30.024070 25786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:43765 every 8 connection(s)
I20260927 15:16:30.025344 25659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:16:30.030264 25789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:30.028417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.67:33239)
0927 15:16:30.029148 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:30.029366 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":278,"threads_started":1}
W20260927 15:16:30.030891 25787 heartbeater.cc:644] Failed to heartbeat to 127.10.33.126:35165 (0 consecutive failures): Network error: Failed to ping master at 127.10.33.126:35165: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
I20260927 15:16:30.031654 25789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:30.031351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.67:49999)
0927 15:16:30.031484 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:30.031549 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 15:16:30.032323 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 25659
I20260927 15:16:30.032526 25789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:30.032170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.67:40449)
0927 15:16:30.032308 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:30.032396 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 15:16:30.032821 25787 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 15:16:30.138394 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:30.137459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.65:40903)
0927 15:16:30.138023 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:30.138157 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":289,"threads_started":1}
I20260927 15:16:30.574239 25792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:30.573293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.66:34847)
0927 15:16:30.573842 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:30.573996 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":256,"threads_started":1}
I20260927 15:16:31.034267 25793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:31.033416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.67:47963)
0927 15:16:31.033923 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:31.034028 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":226,"threads_started":1}
I20260927 15:16:31.140448 25794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:31.139503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.65:39597)
0927 15:16:31.140115 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:31.140221 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":358,"threads_started":1}
I20260927 15:16:31.576119 25795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:31.575342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.66:38443)
0927 15:16:31.575857 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:31.575944 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":268,"threads_started":1}
I20260927 15:16:32.036049 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:32.035242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.67:37263)
0927 15:16:32.035778 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:32.035873 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":261,"threads_started":1}
I20260927 15:16:32.142155 25797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:32.141435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.65:42879)
0927 15:16:32.141882 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:32.141978 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":218,"threads_started":1}
I20260927 15:16:32.577939 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:32.577146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.66:40401)
0927 15:16:32.577627 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:32.577720 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":254,"threads_started":1}
I20260927 15:16:33.037705 25799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:33.036984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.67:54207)
0927 15:16:33.037418 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:33.037504 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":220,"threads_started":1}
I20260927 15:16:33.144050 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:33.143187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.65:57975)
0927 15:16:33.143751 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:33.143849 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":248,"threads_started":1}
I20260927 15:16:33.579674 25801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:33.578885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.66:51117)
0927 15:16:33.579369 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:33.579473 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":229,"threads_started":1}
I20260927 15:16:34.039515 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:34.038741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.67:59791)
0927 15:16:34.039228 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:34.039323 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":237,"threads_started":1}
I20260927 15:16:34.145939 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:34.145077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.65:34011)
0927 15:16:34.145565 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:34.145669 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":257,"threads_started":1}
I20260927 15:16:34.581652 25804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:34.580832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.66:39285)
0927 15:16:34.581373 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:34.581471 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":273,"threads_started":1}
I20260927 15:16:35.041249 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:35.040500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.67:56167)
0927 15:16:35.040982 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:35.041074 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":246,"threads_started":1}
I20260927 15:16:35.147825 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:35.147004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.65:59837)
0927 15:16:35.147554 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:35.147656 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":285,"threads_started":1}
I20260927 15:16:35.583463 25807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:35.582671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.66:53815)
0927 15:16:35.583176 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:35.583268 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":283,"threads_started":1}
I20260927 15:16:36.043015 25808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:36.042294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.67:53135)
0927 15:16:36.042750 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:36.042848 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":224,"threads_started":1}
I20260927 15:16:36.149550 25809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:36.148756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.65:42607)
0927 15:16:36.149248 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:36.149333 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":209,"threads_started":1}
I20260927 15:16:36.585405 25810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:36.584537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.66:41199)
0927 15:16:36.585071 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:36.585170 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":250,"threads_started":1}
I20260927 15:16:37.044939 25811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:37.043974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.67:37083)
0927 15:16:37.044573 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:37.044689 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":277,"threads_started":1}
I20260927 15:16:37.151513 25812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:37.150693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.65:34915)
0927 15:16:37.151214 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:37.151308 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":274,"threads_started":1}
I20260927 15:16:37.587250 25813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:37.586467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.66:56989)
0927 15:16:37.586973 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:37.587063 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":261,"threads_started":1}
I20260927 15:16:38.046720 25814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:38.045998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.67:47711)
0927 15:16:38.046473 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:38.046560 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":244,"threads_started":1}
I20260927 15:16:38.153292 25815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:38.152558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.65:59377)
0927 15:16:38.153036 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:38.153118 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":221,"threads_started":1}
I20260927 15:16:38.589265 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:38.588397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.66:44511)
0927 15:16:38.588984 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:38.589076 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":260,"threads_started":1}
I20260927 15:16:39.048437 25817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:39.047628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.67:51911)
0927 15:16:39.048152 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:39.048264 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":282,"threads_started":1}
I20260927 15:16:39.155225 25818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:39.154444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.65:59991)
0927 15:16:39.154957 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:39.155051 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":254,"threads_started":1}
I20260927 15:16:39.590967 25819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:39.590178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.66:42691)
0927 15:16:39.590662 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:39.590747 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":275,"threads_started":1}
I20260927 15:16:40.050395 25820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.049511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.67:55675)
0927 15:16:40.050088 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:40.050214 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":347,"threads_started":1}
I20260927 15:16:40.157136 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.156287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.65:54277)
0927 15:16:40.156774 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:40.156882 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":239,"threads_started":1}
I20260927 15:16:40.592706 25822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.591974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.66:34703)
0927 15:16:40.592427 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:40.592520 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":212,"threads_started":1}
I20260927 15:16:41.052052 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:41.051333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.67:42963)
0927 15:16:41.051767 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:41.051862 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":204,"threads_started":1}
I20260927 15:16:41.159106 25824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:41.158267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.65:56959)
0927 15:16:41.158774 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:41.158897 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":231,"threads_started":1}
I20260927 15:16:41.594358 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:41.593653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.66:33337)
0927 15:16:41.594110 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:41.594194 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":247,"threads_started":1}
I20260927 15:16:42.053736 25826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.053021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.67:55785)
0927 15:16:42.053486 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:42.053573 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":227,"threads_started":1}
I20260927 15:16:42.161067 25827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.160193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.65:40845)
0927 15:16:42.160758 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:42.160877 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":259,"threads_started":1}
I20260927 15:16:42.596324 25828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.595382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.66:55873)
0927 15:16:42.595984 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:42.596098 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":260,"threads_started":1}
I20260927 15:16:43.055554 25829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:43.054716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.67:32821)
0927 15:16:43.055267 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:43.055363 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":294,"threads_started":1}
I20260927 15:16:43.162820 25830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:43.162074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.65:43015)
0927 15:16:43.162542 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:43.162634 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":214,"threads_started":1}
I20260927 15:16:43.598120 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:43.597315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.66:59713)
0927 15:16:43.597776 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:43.597935 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":215,"threads_started":1}
I20260927 15:16:44.057186 25832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:44.056454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.67:46825)
0927 15:16:44.056911 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:44.057011 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":246,"threads_started":1}
I20260927 15:16:44.164973 25833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:44.164043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.65:57441)
0927 15:16:44.164525 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:44.164653 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":236,"threads_started":1}
I20260927 15:16:44.600157 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:44.599313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.66:39415)
0927 15:16:44.599823 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:44.599928 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":270,"threads_started":1}
I20260927 15:16:45.034976 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 25363
I20260927 15:16:45.053913 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:45.036327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32910 (local address 127.10.33.65:45281)
0927 15:16:45.036961 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:45.036976 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:45.037015 (+ 39us) server_negotiation.cc:407] Connection header received
0927 15:16:45.039025 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:45.039036 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:45.039477 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:45.039774 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:45.041284 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.042624 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.043962 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.044680 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.044878 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:45.047780 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:45.047789 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:45.047807 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:45.047852 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:45.050831 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:45.051563 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:45.051570 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:45.051576 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:45.051743 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:45.052087 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:45.052091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:45.052093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:45.052627 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:45.052855 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:45.053189 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 15:16:45.053590 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":341,"threads_started":1}
I20260927 15:16:45.055249 25520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:45.059268 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:45.058416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:35165 (local address 127.10.33.67:55437)
0927 15:16:45.058920 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:45.059037 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":249,"threads_started":1}
I20260927 15:16:45.141790 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 25363
I20260927 15:16:45.156800 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 25528
I20260927 15:16:45.176157 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:45.157702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59250 (local address 127.10.33.66:43741)
0927 15:16:45.158248 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:45.158262 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:45.158300 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:16:45.160071 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:45.160081 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:45.160527 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:45.160891 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:45.162687 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.164001 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.165321 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.166115 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.166301 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:45.169167 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:45.169176 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:45.169193 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:45.169243 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:45.172545 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:45.173341 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:45.173350 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:45.173358 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:45.173574 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:45.174008 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:45.174012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:45.174014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:45.174592 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:45.174913 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:45.175399 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 15:16:45.175848 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":270,"threads_started":1}
I20260927 15:16:45.177678 25651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:45.263288 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 25528
I20260927 15:16:45.277926 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 25659
I20260927 15:16:45.297092 25840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:45.278905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39034 (local address 127.10.33.67:43765)
0927 15:16:45.279432 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:45.279447 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:45.279489 (+ 42us) server_negotiation.cc:407] Connection header received
0927 15:16:45.281128 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:45.281140 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:45.281583 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:45.282195 (+ 612us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:45.283454 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.284783 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.286219 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.286959 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.287116 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:45.290140 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:45.290149 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:45.290167 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:45.290213 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:45.293420 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:45.294365 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:45.294373 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:45.294381 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:45.294618 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:45.295059 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:45.295062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:45.295064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:45.295671 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:45.295929 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:45.296340 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 15:16:45.296788 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":204,"threads_started":1}
I20260927 15:16:45.298596 25782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:45.383474 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 25659
2026-09-27T15:16:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24401 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:16:45 dist-test-slave-f5h6 krb5kdc[25844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:16:45 dist-test-slave-f5h6 krb5kdc[25844](info): set up 2 sockets
Sep 27 15:16:45 dist-test-slave-f5h6 krb5kdc[25844](info): commencing operation
krb5kdc: starting...
W20260927 15:16:46.410305 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:16:46 dist-test-slave-f5h6 krb5kdc[25844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:16:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:16:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:16:46.646844 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:39747
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:39747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:46.981520 25857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:46.981840 25857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:46.981997 25857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:46.989540 25857 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 15:16:46.989650 25857 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 15:16:46.989684 25857 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 15:16:46.989732 25857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:46.989784 25857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:46.989881 25857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:46.989928 25857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:46.989979 25857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:16:47.000146 25857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:39747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:39747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:47.001453 25857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:47.002751 25857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:47.014631 25863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:47.014690 25862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:47.014667 25865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:47.015455 25857 server_base.cc:1034] running on GCE node
I20260927 15:16:47.016294 25857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:47.017808 25857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:47.019093 25857 hybrid_clock.cc:648] HybridClock initialized: now 1790522207019030 us; error 35 us; skew 500 ppm
I20260927 15:16:47.019450 25857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:47 dist-test-slave-f5h6 krb5kdc[25844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522207, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:47.027101 25857 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:16:47.028889 25857 webserver.cc:460] Webserver started at http://127.10.33.126:42369/ using document root <none> and password file <none>
I20260927 15:16:47.029625 25857 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:47.029716 25857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:47.030121 25857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:47.033525 25857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "23b79be44f5247afa40b92844cbc226b"
format_stamp: "Formatted at 2026-09-27 15:16:47 on dist-test-slave-f5h6"
server_key: "d25cf3b488396fc22d1a9528f65f5641"
server_key_iv: "e0892c4c47db89621c56fffa0cf1e2b4"
server_key_version: "encryptionkey@0"
I20260927 15:16:47.034453 25857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "23b79be44f5247afa40b92844cbc226b"
format_stamp: "Formatted at 2026-09-27 15:16:47 on dist-test-slave-f5h6"
server_key: "d25cf3b488396fc22d1a9528f65f5641"
server_key_iv: "e0892c4c47db89621c56fffa0cf1e2b4"
server_key_version: "encryptionkey@0"
I20260927 15:16:47.039923 25857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 15:16:47.044009 25872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:47.045796 25857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:16:47.046027 25857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "23b79be44f5247afa40b92844cbc226b"
format_stamp: "Formatted at 2026-09-27 15:16:47 on dist-test-slave-f5h6"
server_key: "d25cf3b488396fc22d1a9528f65f5641"
server_key_iv: "e0892c4c47db89621c56fffa0cf1e2b4"
server_key_version: "encryptionkey@0"
I20260927 15:16:47.046277 25857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:47.066502 25857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:47.071128 25857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:47.071393 25857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:47.089675 25857 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:39747
I20260927 15:16:47.089684 25923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:39747 every 8 connection(s)
I20260927 15:16:47.091089 25857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:16:47.094913 25924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:47.100479 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 25857
I20260927 15:16:47.100663 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:16:47.101322 10373 external_mini_cluster.cc:1428] Setting key f876d99ea21345e80730bf02dc757c6b
I20260927 15:16:47.106026 25924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b: Bootstrap starting.
I20260927 15:16:47.109759 25924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:47.110968 25924 log.cc:824] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:47.114048 25924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b: No bootstrap required, opened a new log
Sep 27 15:16:47 dist-test-slave-f5h6 krb5kdc[25844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:16:47.123467 25924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b79be44f5247afa40b92844cbc226b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39747 } }
I20260927 15:16:47.124006 25924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:47.124130 25924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b79be44f5247afa40b92844cbc226b, State: Initialized, Role: FOLLOWER
I20260927 15:16:47.124866 25924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b79be44f5247afa40b92844cbc226b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39747 } }
I20260927 15:16:47.125154 25924 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:16:47.125293 25924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:16:47.125452 25924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:47.126324 25927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:47.103346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38708 (local address 127.10.33.126:39747)
0927 15:16:47.104214 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:47.104226 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:47.104266 (+ 40us) server_negotiation.cc:407] Connection header received
0927 15:16:47.106032 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:47.106055 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:47.106463 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:47.106784 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:47.108727 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:47.110185 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:47.111918 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:47.112679 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:47.112807 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:47.117732 (+ 4925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:47.117741 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:47.117757 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:47.117795 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:47.121094 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:47.122043 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:47.122051 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:47.122058 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:47.122286 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:47.122877 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:47.122901 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:47.122904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:47.123822 (+ 918us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:47.124123 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:47.124645 (+ 522us) server_negotiation.cc:299] Negotiation successful
0927 15:16:47.125059 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":388,"threads_started":1}
I20260927 15:16:47.128127 25924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b79be44f5247afa40b92844cbc226b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39747 } }
I20260927 15:16:47.128625 25924 leader_election.cc:302] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23b79be44f5247afa40b92844cbc226b; no voters:
I20260927 15:16:47.129276 25924 leader_election.cc:288] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:16:47.129398 25929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:47.130106 25929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 1 LEADER]: Becoming Leader. State: Replica: 23b79be44f5247afa40b92844cbc226b, State: Running, Role: LEADER
I20260927 15:16:47.130759 25929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b79be44f5247afa40b92844cbc226b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39747 } }
I20260927 15:16:47.131500 25924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:47.133543 25931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23b79be44f5247afa40b92844cbc226b. Latest consensus state: current_term: 1 leader_uuid: "23b79be44f5247afa40b92844cbc226b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b79be44f5247afa40b92844cbc226b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39747 } } }
I20260927 15:16:47.133528 25930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23b79be44f5247afa40b92844cbc226b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b79be44f5247afa40b92844cbc226b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39747 } } }
I20260927 15:16:47.133968 25931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:47.133965 25930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:47.135939 25935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:16:47.140625 25935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:16:47.150239 25935 catalog_manager.cc:1347] Generated new cluster ID: eb15d132df1841e187802297a62c16de
I20260927 15:16:47.150354 25935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:16:47.181185 25935 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:16:47.182579 25935 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:16:47.192845 25935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b: Generated new TSK 0
I20260927 15:16:47.194165 25935 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.33.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
I20260927 15:16:47.262007 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:39747
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:47.590655 25950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:47.590950 25950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:47.591087 25950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:47.598359 25950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:47.598491 25950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:47.598645 25950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:16:47.608345 25950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:39747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:47.609655 25950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:47.610947 25950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:47.623546 25955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:47.623559 25958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:47.623811 25956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:47.625272 25950 server_base.cc:1034] running on GCE node
I20260927 15:16:47.626034 25950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:47.627121 25950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:47.628346 25950 hybrid_clock.cc:648] HybridClock initialized: now 1790522207628291 us; error 43 us; skew 500 ppm
I20260927 15:16:47.628659 25950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:47 dist-test-slave-f5h6 krb5kdc[25844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522207, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:47.631771 25950 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:16:47.633199 25950 webserver.cc:460] Webserver started at http://127.10.33.65:45933/ using document root <none> and password file <none>
I20260927 15:16:47.633829 25950 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:47.633984 25950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:47.634274 25950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:47.637271 25950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "64e324de2d014f7980bf068d1cf49792"
format_stamp: "Formatted at 2026-09-27 15:16:47 on dist-test-slave-f5h6"
server_key: "c364214df20d066e5ebf77b0b1d766e8"
server_key_iv: "c464ba63cefedcd00bc822a8213c42f9"
server_key_version: "encryptionkey@0"
I20260927 15:16:47.638058 25950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "64e324de2d014f7980bf068d1cf49792"
format_stamp: "Formatted at 2026-09-27 15:16:47 on dist-test-slave-f5h6"
server_key: "c364214df20d066e5ebf77b0b1d766e8"
server_key_iv: "c464ba63cefedcd00bc822a8213c42f9"
server_key_version: "encryptionkey@0"
I20260927 15:16:47.643674 25950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260927 15:16:47.647848 25965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:47.649636 25950 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 15:16:47.649910 25950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "64e324de2d014f7980bf068d1cf49792"
format_stamp: "Formatted at 2026-09-27 15:16:47 on dist-test-slave-f5h6"
server_key: "c364214df20d066e5ebf77b0b1d766e8"
server_key_iv: "c464ba63cefedcd00bc822a8213c42f9"
server_key_version: "encryptionkey@0"
I20260927 15:16:47.650179 25950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:47.666689 25950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:47.671473 25950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:47.671819 25950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:47.672923 25950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:47.674615 25950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:47.674710 25950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:47.674862 25950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:47.674926 25950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:47.708536 25950 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:33097
I20260927 15:16:47.708570 26077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:33097 every 8 connection(s)
I20260927 15:16:47.709913 25950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:16:47.717548 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 25950
I20260927 15:16:47.717757 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:16:47.718425 10373 external_mini_cluster.cc:1428] Setting key e94e0b67d8272c4474955d9a9bfd4cc2
Sep 27 15:16:47 dist-test-slave-f5h6 krb5kdc[25844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522207, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:16:47.731559 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:47.712391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:44039 (local address 127.10.33.126:39747)
0927 15:16:47.713844 (+ 1453us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:47.713850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:47.714132 (+ 282us) server_negotiation.cc:407] Connection header received
0927 15:16:47.715939 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:47.715944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:47.716025 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:47.716131 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:47.717808 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:47.718685 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:47.720608 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:47.721023 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:47.721141 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:47.725043 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:47.725058 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:47.725064 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:47.725114 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:47.728126 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:47.729059 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:47.729065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:47.729069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:47.729221 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:47.729623 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:47.729628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:47.729632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:47.730016 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:47.730154 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:47.730923 (+ 769us) server_negotiation.cc:299] Negotiation successful
0927 15:16:47.731172 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1309,"thread_start_us":1161,"threads_started":1}
I20260927 15:16:47.732601 26081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:47.712957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.65:44039)
0927 15:16:47.713971 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:47.714016 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:47.715664 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:47.716387 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:47.716397 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:47.716936 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:47.717586 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:47.717606 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:47.718866 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:47.718870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:47.720459 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:47.720475 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:47.721260 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:47.721268 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:47.721527 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:47.722226 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:47.722252 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:47.724702 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:47.728333 (+ 3631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:47.728341 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:47.728355 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:47.728920 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:47.729330 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:47.729333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:47.729336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:47.729530 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:47.730159 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:47.730165 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:47.730527 (+ 362us) client_negotiation.cc:769] Sending connection context
0927 15:16:47.730786 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 15:16:47.731262 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":443,"threads_started":1}
I20260927 15:16:47.734450 26078 heartbeater.cc:347] Connected to a master server at 127.10.33.126:39747
I20260927 15:16:47.734808 26078 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:47.735721 26078 heartbeater.cc:511] Master 127.10.33.126:39747 requested a full tablet report, sending...
I20260927 15:16:47.738461 25889 ts_manager.cc:194] Registered new tserver with Master: 64e324de2d014f7980bf068d1cf49792 (127.10.33.65:33097)
I20260927 15:16:47.740579 25889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:44039
WARNING: no policy specified for kudu/127.10.33.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
I20260927 15:16:47.773701 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:39747
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:48.100106 26084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:48.100383 26084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:48.100521 26084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:48.108009 26084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:48.108155 26084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:48.108304 26084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:16:48.118392 26084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:39747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:48.119706 26084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:48.121037 26084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:48.134115 26089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:48.134141 26092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:48.134174 26090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:48.135159 26084 server_base.cc:1034] running on GCE node
I20260927 15:16:48.135828 26084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:48.136845 26084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:48.138055 26084 hybrid_clock.cc:648] HybridClock initialized: now 1790522208137959 us; error 55 us; skew 500 ppm
I20260927 15:16:48.138379 26084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:48 dist-test-slave-f5h6 krb5kdc[25844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522208, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:48.141848 26084 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:16:48.143497 26084 webserver.cc:460] Webserver started at http://127.10.33.66:36465/ using document root <none> and password file <none>
I20260927 15:16:48.144253 26084 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:48.144383 26084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:48.144728 26084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:48.148018 26084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "ccece2bca11944899ae41baa61d49801"
format_stamp: "Formatted at 2026-09-27 15:16:48 on dist-test-slave-f5h6"
server_key: "d0920b366530a4e93fbfeb89a670ceee"
server_key_iv: "2e7bc8d665cd29f159ef31ca959ca4fb"
server_key_version: "encryptionkey@0"
I20260927 15:16:48.148869 26084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "ccece2bca11944899ae41baa61d49801"
format_stamp: "Formatted at 2026-09-27 15:16:48 on dist-test-slave-f5h6"
server_key: "d0920b366530a4e93fbfeb89a670ceee"
server_key_iv: "2e7bc8d665cd29f159ef31ca959ca4fb"
server_key_version: "encryptionkey@0"
I20260927 15:16:48.154875 26084 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 15:16:48.158996 26099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:48.160679 26084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:16:48.160861 26084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "ccece2bca11944899ae41baa61d49801"
format_stamp: "Formatted at 2026-09-27 15:16:48 on dist-test-slave-f5h6"
server_key: "d0920b366530a4e93fbfeb89a670ceee"
server_key_iv: "2e7bc8d665cd29f159ef31ca959ca4fb"
server_key_version: "encryptionkey@0"
I20260927 15:16:48.161063 26084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:48.176179 26084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:48.180546 26084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:48.180797 26084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:48.181845 26084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:48.183229 26084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:48.183305 26084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:48.183389 26084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:48.183444 26084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:48.216082 26084 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:35461
I20260927 15:16:48.216097 26211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:35461 every 8 connection(s)
I20260927 15:16:48.217464 26084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:16:48.226413 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 26084
I20260927 15:16:48.226621 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:16:48.227064 10373 external_mini_cluster.cc:1428] Setting key fab8211c4f1a8ec31595c1a38c5ae4c4
Sep 27 15:16:48 dist-test-slave-f5h6 krb5kdc[25844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522208, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:16:48.239122 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:48.220125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:41663 (local address 127.10.33.126:39747)
0927 15:16:48.220300 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:48.220305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:48.221570 (+ 1265us) server_negotiation.cc:407] Connection header received
0927 15:16:48.223459 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:48.223466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:48.223551 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:48.223647 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:48.225222 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.225993 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.227712 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.228332 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.228476 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:48.232649 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:48.232654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:48.232658 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:48.232696 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:48.235537 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:48.236462 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:48.236469 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:48.236472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:48.236641 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:48.237119 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:48.237125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:48.237128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:48.237523 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:48.237690 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:48.238531 (+ 841us) server_negotiation.cc:299] Negotiation successful
0927 15:16:48.238780 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:16:48.239988 26214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:48.220601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.66:41663)
0927 15:16:48.221392 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:48.221432 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:48.223164 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:48.223891 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:48.223899 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:48.224437 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:48.225035 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:48.225054 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.226167 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.226171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:48.227549 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:48.227575 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.228599 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.228604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:48.228849 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:48.229589 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:48.229621 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:48.232370 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:48.235707 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:48.235717 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:48.235738 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:48.236322 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:48.236750 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:48.236754 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:48.236757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:48.237005 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:48.237683 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:48.237690 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:48.238151 (+ 461us) client_negotiation.cc:769] Sending connection context
0927 15:16:48.238397 (+ 246us) client_negotiation.cc:240] Negotiation successful
0927 15:16:48.238829 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":345,"threads_started":1}
I20260927 15:16:48.241472 26212 heartbeater.cc:347] Connected to a master server at 127.10.33.126:39747
I20260927 15:16:48.241837 26212 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:48.242805 26212 heartbeater.cc:511] Master 127.10.33.126:39747 requested a full tablet report, sending...
I20260927 15:16:48.244477 25889 ts_manager.cc:194] Registered new tserver with Master: ccece2bca11944899ae41baa61d49801 (127.10.33.66:35461)
I20260927 15:16:48.245316 25889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:41663
WARNING: no policy specified for kudu/127.10.33.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
I20260927 15:16:48.282254 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:39747
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:48.608134 26217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:48.608420 26217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:48.608542 26217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:48.615942 26217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:48.616046 26217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:48.616190 26217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:16:48.625959 26217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:39747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:16:48.627262 26217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:48.628607 26217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:48.641732 26223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:48.641781 26222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:48.641808 26225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:48.642599 26217 server_base.cc:1034] running on GCE node
I20260927 15:16:48.643277 26217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:48.644261 26217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:48.645499 26217 hybrid_clock.cc:648] HybridClock initialized: now 1790522208645440 us; error 50 us; skew 500 ppm
I20260927 15:16:48.645882 26217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:16:48 dist-test-slave-f5h6 krb5kdc[25844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522208, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:48.649257 26217 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:16:48.650842 26217 webserver.cc:460] Webserver started at http://127.10.33.67:33701/ using document root <none> and password file <none>
I20260927 15:16:48.651571 26217 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:48.651688 26217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:48.651986 26217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:48.655071 26217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "058c7d5d33ad46af8d87df79504209c5"
format_stamp: "Formatted at 2026-09-27 15:16:48 on dist-test-slave-f5h6"
server_key: "e29ea2a1e4c62e9961ad206000a5d107"
server_key_iv: "5cc9d537c274292370bc0518228972ac"
server_key_version: "encryptionkey@0"
I20260927 15:16:48.655795 26217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "058c7d5d33ad46af8d87df79504209c5"
format_stamp: "Formatted at 2026-09-27 15:16:48 on dist-test-slave-f5h6"
server_key: "e29ea2a1e4c62e9961ad206000a5d107"
server_key_iv: "5cc9d537c274292370bc0518228972ac"
server_key_version: "encryptionkey@0"
I20260927 15:16:48.660840 26217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:16:48.664649 26232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:48.666281 26217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:16:48.666460 26217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "058c7d5d33ad46af8d87df79504209c5"
format_stamp: "Formatted at 2026-09-27 15:16:48 on dist-test-slave-f5h6"
server_key: "e29ea2a1e4c62e9961ad206000a5d107"
server_key_iv: "5cc9d537c274292370bc0518228972ac"
server_key_version: "encryptionkey@0"
I20260927 15:16:48.666651 26217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:48.678870 26217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:48.683099 26217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:48.683373 26217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:48.684374 26217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:48.685662 26217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:48.685747 26217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:48.685907 26217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:48.685968 26217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 15:16:48.717989 26217 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:32889
I20260927 15:16:48.718008 26344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:32889 every 8 connection(s)
I20260927 15:16:48.719434 26217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:16:48.724756 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 26217
I20260927 15:16:48.724936 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:16:48.725471 10373 external_mini_cluster.cc:1428] Setting key c8b4888bceec04b34b870a4a2a8ffb2d
Sep 27 15:16:48 dist-test-slave-f5h6 krb5kdc[25844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522208, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:16:48.740453 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:48.722005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:39193 (local address 127.10.33.126:39747)
0927 15:16:48.722199 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:48.722203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:48.723308 (+ 1105us) server_negotiation.cc:407] Connection header received
0927 15:16:48.725231 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:48.725239 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:48.725338 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:48.725448 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:48.727262 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.728186 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.730025 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.730528 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.730641 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:48.734381 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:48.734386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:48.734390 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:48.734429 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:48.736990 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:48.737773 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:48.737777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:48.737779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:48.737997 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:48.738417 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:48.738421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:48.738423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:48.738729 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:48.738886 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:48.739854 (+ 968us) server_negotiation.cc:299] Negotiation successful
0927 15:16:48.740120 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:16:48.741514 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:48.722394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.67:39193)
0927 15:16:48.723139 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:48.723184 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:48.724907 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:48.725711 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:48.725720 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:48.726434 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:48.727074 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:48.727097 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.728351 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.728356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:48.729722 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:48.729741 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.730778 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.730784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:48.731057 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:48.731786 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:48.731821 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:48.734122 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:48.737161 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:48.737169 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:48.737184 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:48.737668 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:48.738089 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:48.738092 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:48.738094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:48.738303 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:48.738892 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:48.738899 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:48.739366 (+ 467us) client_negotiation.cc:769] Sending connection context
0927 15:16:48.739676 (+ 310us) client_negotiation.cc:240] Negotiation successful
0927 15:16:48.740246 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":278,"threads_started":1}
I20260927 15:16:48.742964 26345 heartbeater.cc:347] Connected to a master server at 127.10.33.126:39747
I20260927 15:16:48.743291 26345 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:48.744087 26345 heartbeater.cc:511] Master 127.10.33.126:39747 requested a full tablet report, sending...
I20260927 15:16:48.744089 26078 heartbeater.cc:503] Master 127.10.33.126:39747 was elected leader, sending a full tablet report...
I20260927 15:16:48.745699 25888 ts_manager.cc:194] Registered new tserver with Master: 058c7d5d33ad46af8d87df79504209c5 (127.10.33.67:32889)
I20260927 15:16:48.746578 25888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:39193
I20260927 15:16:48.753755 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:16:48.772436 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:48.759235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38712 (local address 127.10.33.126:39747)
0927 15:16:48.759413 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:48.759419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:48.759438 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:16:48.759496 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:48.759501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:48.759583 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:48.759681 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:48.761617 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.762498 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.763996 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.764414 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.764511 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:48.766992 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:48.766998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:48.767004 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:48.767041 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:48.769615 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:48.770398 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:48.770404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:48.770408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:48.770575 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:48.770958 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:48.770961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:48.770964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:48.771322 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:48.771557 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:48.771937 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 15:16:48.772122 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:16:48.776671 25888 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:16:48.783463 25888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:16:48.807041 26358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:48.795902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57836 (local address 127.10.33.67:32889)
0927 15:16:48.798604 (+ 2702us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:48.798612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:48.798642 (+ 30us) server_negotiation.cc:407] Connection header received
0927 15:16:48.798843 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:48.798852 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:48.799115 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:48.800128 (+ 1013us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:48.800416 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.801982 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.804390 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.805842 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.806009 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:48.806275 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:48.806518 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 15:16:48.806698 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2516,"thread_start_us":2373,"threads_started":1}
I20260927 15:16:48.807193 26357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:48.795598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:32889 (local address 127.0.0.1:57836)
0927 15:16:48.796471 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:48.796524 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:48.796720 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:48.799383 (+ 2663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:48.799391 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:48.799422 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:48.799908 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:48.799922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.802101 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.802106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:48.804198 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:48.804214 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.806060 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.806067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:48.806267 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:48.806332 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 15:16:48.806458 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 15:16:48.806681 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":116,"thread_start_us":874,"threads_started":1}
I20260927 15:16:48.807736 26361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:48.797379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:35461 (local address 127.0.0.1:50322)
0927 15:16:48.799395 (+ 2016us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:48.799420 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:48.799553 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:48.800193 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:48.800197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:48.800223 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:48.800610 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:48.800623 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.802377 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.802380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:48.804823 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:48.804839 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.806921 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.806927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:48.807088 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:48.807115 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 15:16:48.807196 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 15:16:48.807316 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1889,"mutex_wait_us":55,"spinlock_wait_cycles":768,"thread_start_us":1746,"threads_started":1}
I20260927 15:16:48.808090 26360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:48.796501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49818 (local address 127.10.33.65:33097)
0927 15:16:48.797355 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:48.797366 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:48.798948 (+ 1582us) server_negotiation.cc:407] Connection header received
0927 15:16:48.799044 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:48.799051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:48.799230 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:48.799436 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:48.800196 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.801746 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.805015 (+ 3269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.806461 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.807284 (+ 823us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:48.807526 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:48.807616 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 15:16:48.807786 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":559,"threads_started":1}
I20260927 15:16:48.808223 26362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:48.797619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50322 (local address 127.10.33.66:35461)
0927 15:16:48.798957 (+ 1338us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:48.798968 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:48.799682 (+ 714us) server_negotiation.cc:407] Connection header received
0927 15:16:48.799782 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:48.799789 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:48.800012 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:48.800173 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:48.800790 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.802238 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.805015 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.806471 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.806595 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:48.806783 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:48.807727 (+ 944us) server_negotiation.cc:299] Negotiation successful
0927 15:16:48.807913 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1156,"thread_start_us":1099,"threads_started":1}
I20260927 15:16:48.808573 26359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:48.796219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33097 (local address 127.0.0.1:49818)
0927 15:16:48.797890 (+ 1671us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:48.797913 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:48.798032 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:48.799453 (+ 1421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:48.799458 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:48.799494 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:48.799999 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:48.800009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.801944 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.801950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:48.804823 (+ 2873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:48.804841 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.806698 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.806705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:48.806896 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:48.806931 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 15:16:48.806996 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 15:16:48.807131 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1505,"mutex_wait_us":216,"spinlock_wait_cycles":14720,"thread_start_us":1530,"threads_started":1}
I20260927 15:16:48.811580 26147 tablet_service.cc:1467] Processing CreateTablet for tablet e0665b6bce174441bc5c6116daa3112f (DEFAULT_TABLE table=test-table [id=2ca8e9d6230a482686000a0a3e7c9396]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:48.811619 26280 tablet_service.cc:1467] Processing CreateTablet for tablet e0665b6bce174441bc5c6116daa3112f (DEFAULT_TABLE table=test-table [id=2ca8e9d6230a482686000a0a3e7c9396]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:48.811793 26013 tablet_service.cc:1467] Processing CreateTablet for tablet e0665b6bce174441bc5c6116daa3112f (DEFAULT_TABLE table=test-table [id=2ca8e9d6230a482686000a0a3e7c9396]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:48.812914 26147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0665b6bce174441bc5c6116daa3112f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:48.812997 26280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0665b6bce174441bc5c6116daa3112f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:48.812999 26013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0665b6bce174441bc5c6116daa3112f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:48.825027 26363 tablet_bootstrap.cc:492] T e0665b6bce174441bc5c6116daa3112f P ccece2bca11944899ae41baa61d49801: Bootstrap starting.
I20260927 15:16:48.827592 26364 tablet_bootstrap.cc:492] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792: Bootstrap starting.
I20260927 15:16:48.829828 26363 tablet_bootstrap.cc:654] T e0665b6bce174441bc5c6116daa3112f P ccece2bca11944899ae41baa61d49801: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:48.830644 26365 tablet_bootstrap.cc:492] T e0665b6bce174441bc5c6116daa3112f P 058c7d5d33ad46af8d87df79504209c5: Bootstrap starting.
I20260927 15:16:48.831555 26363 log.cc:824] T e0665b6bce174441bc5c6116daa3112f P ccece2bca11944899ae41baa61d49801: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:48.832476 26364 tablet_bootstrap.cc:654] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:48.833752 26364 log.cc:824] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:48.834082 26365 tablet_bootstrap.cc:654] T e0665b6bce174441bc5c6116daa3112f P 058c7d5d33ad46af8d87df79504209c5: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:48.835151 26363 tablet_bootstrap.cc:492] T e0665b6bce174441bc5c6116daa3112f P ccece2bca11944899ae41baa61d49801: No bootstrap required, opened a new log
I20260927 15:16:48.835327 26365 log.cc:824] T e0665b6bce174441bc5c6116daa3112f P 058c7d5d33ad46af8d87df79504209c5: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:48.835464 26363 ts_tablet_manager.cc:1397] T e0665b6bce174441bc5c6116daa3112f P ccece2bca11944899ae41baa61d49801: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260927 15:16:48.837273 26364 tablet_bootstrap.cc:492] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792: No bootstrap required, opened a new log
I20260927 15:16:48.837652 26364 ts_tablet_manager.cc:1397] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260927 15:16:48.844614 26365 tablet_bootstrap.cc:492] T e0665b6bce174441bc5c6116daa3112f P 058c7d5d33ad46af8d87df79504209c5: No bootstrap required, opened a new log
I20260927 15:16:48.844930 26365 ts_tablet_manager.cc:1397] T e0665b6bce174441bc5c6116daa3112f P 058c7d5d33ad46af8d87df79504209c5: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.011s
I20260927 15:16:48.846884 26363 raft_consensus.cc:348] T e0665b6bce174441bc5c6116daa3112f P ccece2bca11944899ae41baa61d49801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 } } peers { permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 } } peers { permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 } }
I20260927 15:16:48.847433 26363 raft_consensus.cc:374] T e0665b6bce174441bc5c6116daa3112f P ccece2bca11944899ae41baa61d49801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:48.847627 26363 raft_consensus.cc:729] T e0665b6bce174441bc5c6116daa3112f P ccece2bca11944899ae41baa61d49801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccece2bca11944899ae41baa61d49801, State: Initialized, Role: FOLLOWER
I20260927 15:16:48.848395 26363 consensus_queue.cc:260] T e0665b6bce174441bc5c6116daa3112f P ccece2bca11944899ae41baa61d49801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 } } peers { permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 } } peers { permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 } }
I20260927 15:16:48.848810 26364 raft_consensus.cc:348] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 } } peers { permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 } } peers { permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 } }
I20260927 15:16:48.849378 26364 raft_consensus.cc:374] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:48.849540 26364 raft_consensus.cc:729] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64e324de2d014f7980bf068d1cf49792, State: Initialized, Role: FOLLOWER
I20260927 15:16:48.850322 26364 consensus_queue.cc:260] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 } } peers { permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 } } peers { permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 } }
I20260927 15:16:48.850625 26212 heartbeater.cc:503] Master 127.10.33.126:39747 was elected leader, sending a full tablet report...
I20260927 15:16:48.852574 26364 ts_tablet_manager.cc:1428] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260927 15:16:48.854105 26363 ts_tablet_manager.cc:1428] T e0665b6bce174441bc5c6116daa3112f P ccece2bca11944899ae41baa61d49801: Time spent starting tablet: real 0.018s user 0.005s sys 0.010s
I20260927 15:16:48.854202 26365 raft_consensus.cc:348] T e0665b6bce174441bc5c6116daa3112f P 058c7d5d33ad46af8d87df79504209c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 } } peers { permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 } } peers { permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 } }
I20260927 15:16:48.854687 26365 raft_consensus.cc:374] T e0665b6bce174441bc5c6116daa3112f P 058c7d5d33ad46af8d87df79504209c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:48.854796 26365 raft_consensus.cc:729] T e0665b6bce174441bc5c6116daa3112f P 058c7d5d33ad46af8d87df79504209c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 058c7d5d33ad46af8d87df79504209c5, State: Initialized, Role: FOLLOWER
I20260927 15:16:48.855495 26365 consensus_queue.cc:260] T e0665b6bce174441bc5c6116daa3112f P 058c7d5d33ad46af8d87df79504209c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 } } peers { permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 } } peers { permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 } }
I20260927 15:16:48.856777 26345 heartbeater.cc:503] Master 127.10.33.126:39747 was elected leader, sending a full tablet report...
I20260927 15:16:48.858717 26365 ts_tablet_manager.cc:1428] T e0665b6bce174441bc5c6116daa3112f P 058c7d5d33ad46af8d87df79504209c5: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260927 15:16:48.962298 26079 tablet.cc:2369] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:16:48.969542 26213 tablet.cc:2369] T e0665b6bce174441bc5c6116daa3112f P ccece2bca11944899ae41baa61d49801: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:16:48.971024 26346 tablet.cc:2369] T e0665b6bce174441bc5c6116daa3112f P 058c7d5d33ad46af8d87df79504209c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:16:49.007982 26370 raft_consensus.cc:482] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:16:49.008294 26370 raft_consensus.cc:504] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 } } peers { permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 } } peers { permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 } }
I20260927 15:16:49.010192 26370 leader_election.cc:288] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 058c7d5d33ad46af8d87df79504209c5 (127.10.33.67:32889), ccece2bca11944899ae41baa61d49801 (127.10.33.66:35461)
I20260927 15:16:49.017254 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:49.010562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:32889 (local address 127.10.33.65:54485)
0927 15:16:49.011188 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:49.011208 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:49.011407 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:49.011801 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:49.011806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:49.011838 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:49.012187 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:49.012199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.013405 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.013411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:49.015100 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:49.015112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.016426 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.016433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:49.016605 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:49.016641 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:16:49.016748 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 15:16:49.016886 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":42,"thread_start_us":343,"threads_started":1}
I20260927 15:16:49.017310 26358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:49.010624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:54485 (local address 127.10.33.67:32889)
0927 15:16:49.010872 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:49.010879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:49.011318 (+ 439us) server_negotiation.cc:407] Connection header received
0927 15:16:49.011563 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:49.011570 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:49.011651 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:49.011737 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:49.012337 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.013277 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.015263 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.016222 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.016341 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:49.016393 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:49.016921 (+ 528us) server_negotiation.cc:299] Negotiation successful
0927 15:16:49.017065 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 15:16:49.017639 26362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:49.011504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:39465 (local address 127.10.33.66:35461)
0927 15:16:49.011708 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:49.011714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:49.012058 (+ 344us) server_negotiation.cc:407] Connection header received
0927 15:16:49.012200 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:49.012206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:49.012283 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:49.012358 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:49.012956 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.013851 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.015750 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.016675 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.016810 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:49.016876 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:49.017276 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 15:16:49.017409 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 15:16:49.017951 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:49.011314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:35461 (local address 127.10.33.65:39465)
0927 15:16:49.011956 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:49.011972 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:49.012083 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:49.012418 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:49.012422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:49.012448 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:49.012809 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:49.012820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.014023 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.014027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:49.015596 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:49.015611 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.016922 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.016929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:49.017099 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:49.017130 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 15:16:49.017245 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 15:16:49.017585 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":1,"thread_start_us":321,"threads_started":1}
I20260927 15:16:49.018493 26300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0665b6bce174441bc5c6116daa3112f" candidate_uuid: "64e324de2d014f7980bf068d1cf49792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058c7d5d33ad46af8d87df79504209c5" is_pre_election: true
I20260927 15:16:49.018733 26167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0665b6bce174441bc5c6116daa3112f" candidate_uuid: "64e324de2d014f7980bf068d1cf49792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccece2bca11944899ae41baa61d49801" is_pre_election: true
I20260927 15:16:49.018981 26300 raft_consensus.cc:2454] T e0665b6bce174441bc5c6116daa3112f P 058c7d5d33ad46af8d87df79504209c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64e324de2d014f7980bf068d1cf49792 in term 0.
I20260927 15:16:49.019197 26167 raft_consensus.cc:2454] T e0665b6bce174441bc5c6116daa3112f P ccece2bca11944899ae41baa61d49801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64e324de2d014f7980bf068d1cf49792 in term 0.
I20260927 15:16:49.019631 25967 leader_election.cc:302] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058c7d5d33ad46af8d87df79504209c5, 64e324de2d014f7980bf068d1cf49792; no voters:
I20260927 15:16:49.020008 26370 raft_consensus.cc:2781] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:16:49.020165 26370 raft_consensus.cc:482] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:16:49.020320 26370 raft_consensus.cc:3037] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:49.022504 26370 raft_consensus.cc:504] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 } } peers { permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 } } peers { permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 } }
I20260927 15:16:49.023105 26370 leader_election.cc:288] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [CANDIDATE]: Term 1 election: Requested vote from peers 058c7d5d33ad46af8d87df79504209c5 (127.10.33.67:32889), ccece2bca11944899ae41baa61d49801 (127.10.33.66:35461)
I20260927 15:16:49.023613 26300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0665b6bce174441bc5c6116daa3112f" candidate_uuid: "64e324de2d014f7980bf068d1cf49792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058c7d5d33ad46af8d87df79504209c5"
I20260927 15:16:49.023624 26167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0665b6bce174441bc5c6116daa3112f" candidate_uuid: "64e324de2d014f7980bf068d1cf49792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccece2bca11944899ae41baa61d49801"
I20260927 15:16:49.023799 26300 raft_consensus.cc:3037] T e0665b6bce174441bc5c6116daa3112f P 058c7d5d33ad46af8d87df79504209c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:49.023813 26167 raft_consensus.cc:3037] T e0665b6bce174441bc5c6116daa3112f P ccece2bca11944899ae41baa61d49801 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:49.025971 26300 raft_consensus.cc:2454] T e0665b6bce174441bc5c6116daa3112f P 058c7d5d33ad46af8d87df79504209c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64e324de2d014f7980bf068d1cf49792 in term 1.
I20260927 15:16:49.026074 26167 raft_consensus.cc:2454] T e0665b6bce174441bc5c6116daa3112f P ccece2bca11944899ae41baa61d49801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64e324de2d014f7980bf068d1cf49792 in term 1.
I20260927 15:16:49.026360 25967 leader_election.cc:302] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058c7d5d33ad46af8d87df79504209c5, 64e324de2d014f7980bf068d1cf49792; no voters:
I20260927 15:16:49.026654 26370 raft_consensus.cc:2781] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:49.027117 26370 raft_consensus.cc:686] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [term 1 LEADER]: Becoming Leader. State: Replica: 64e324de2d014f7980bf068d1cf49792, State: Running, Role: LEADER
I20260927 15:16:49.027586 26370 consensus_queue.cc:237] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 } } peers { permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 } } peers { permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 } }
I20260927 15:16:49.032019 25889 catalog_manager.cc:5495] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64e324de2d014f7980bf068d1cf49792 (127.10.33.65). New cstate: current_term: 1 leader_uuid: "64e324de2d014f7980bf068d1cf49792" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 } health_report { overall_health: HEALTHY } } }
I20260927 15:16:49.048790 26360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:49.043195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.10.33.65:33097)
0927 15:16:49.043395 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:49.043401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:49.043429 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:16:49.043555 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:49.043561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:49.043672 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:49.043802 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:16:49.044224 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.044950 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.046630 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.047058 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.047169 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:49.047679 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:16:49.048199 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:16:49.048332 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:49.048434 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 15:16:49.048535 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:16:49.057821 26167 raft_consensus.cc:1261] T e0665b6bce174441bc5c6116daa3112f P ccece2bca11944899ae41baa61d49801 [term 1 FOLLOWER]: Refusing update from remote peer 64e324de2d014f7980bf068d1cf49792: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:49.057840 26300 raft_consensus.cc:1261] T e0665b6bce174441bc5c6116daa3112f P 058c7d5d33ad46af8d87df79504209c5 [term 1 FOLLOWER]: Refusing update from remote peer 64e324de2d014f7980bf068d1cf49792: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:49.058818 26370 consensus_queue.cc:1035] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [LEADER]: Connected to new peer: Peer: permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:49.059175 26374 consensus_queue.cc:1035] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:49.090525 25889 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:49.090860 25889 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:49.095304 25889 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b: Sending DeleteTablet for 3 replicas of tablet e0665b6bce174441bc5c6116daa3112f
I20260927 15:16:49.096779 26013 tablet_service.cc:1514] Processing DeleteTablet for tablet e0665b6bce174441bc5c6116daa3112f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:49 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:49818
I20260927 15:16:49.097093 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 25950
I20260927 15:16:49.097033 26280 tablet_service.cc:1514] Processing DeleteTablet for tablet e0665b6bce174441bc5c6116daa3112f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:49 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:57836
I20260927 15:16:49.097687 26147 tablet_service.cc:1514] Processing DeleteTablet for tablet e0665b6bce174441bc5c6116daa3112f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:49 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:50322
I20260927 15:16:49.113000 26390 tablet_replica.cc:330] T e0665b6bce174441bc5c6116daa3112f P ccece2bca11944899ae41baa61d49801: stopping tablet replica
I20260927 15:16:49.113585 26390 raft_consensus.cc:2229] T e0665b6bce174441bc5c6116daa3112f P ccece2bca11944899ae41baa61d49801 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:16:49.115686 26390 raft_consensus.cc:2258] T e0665b6bce174441bc5c6116daa3112f P ccece2bca11944899ae41baa61d49801 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:49.119758 26390 ts_tablet_manager.cc:1905] T e0665b6bce174441bc5c6116daa3112f P ccece2bca11944899ae41baa61d49801: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:49.120594 26388 tablet_replica.cc:330] T e0665b6bce174441bc5c6116daa3112f P 058c7d5d33ad46af8d87df79504209c5: stopping tablet replica
I20260927 15:16:49.121217 26388 raft_consensus.cc:2229] T e0665b6bce174441bc5c6116daa3112f P 058c7d5d33ad46af8d87df79504209c5 [term 1 FOLLOWER]: Raft consensus shutting down.
Sep 27 15:16:49 dist-test-slave-f5h6 krb5kdc[25844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.65@KRBTEST.COM
I20260927 15:16:49.121912 26388 raft_consensus.cc:2258] T e0665b6bce174441bc5c6116daa3112f P 058c7d5d33ad46af8d87df79504209c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:49.124368 26388 ts_tablet_manager.cc:1905] T e0665b6bce174441bc5c6116daa3112f P 058c7d5d33ad46af8d87df79504209c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:49.129583 26390 ts_tablet_manager.cc:1918] T e0665b6bce174441bc5c6116daa3112f P ccece2bca11944899ae41baa61d49801: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:16:49.129769 26390 log.cc:1196] T e0665b6bce174441bc5c6116daa3112f P ccece2bca11944899ae41baa61d49801: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/e0665b6bce174441bc5c6116daa3112f
I20260927 15:16:49.130115 26387 tablet_replica.cc:330] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792: stopping tablet replica
I20260927 15:16:49.130420 26390 ts_tablet_manager.cc:1939] T e0665b6bce174441bc5c6116daa3112f P ccece2bca11944899ae41baa61d49801: Deleting consensus metadata
I20260927 15:16:49.130568 26387 raft_consensus.cc:2229] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:16:49.130648 26360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:49.098386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49828 (local address 127.10.33.65:33097)
0927 15:16:49.098588 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:49.098594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:49.109377 (+ 10783us) server_negotiation.cc:407] Connection header received
0927 15:16:49.110086 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:49.110092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:49.110251 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:49.110397 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:49.112717 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.113980 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.116029 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.116793 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.116901 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:49.123071 (+ 6170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:49.123094 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:49.123113 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:49.123160 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:49.126867 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:49.127888 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:49.127897 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:49.127901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:49.128073 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:49.128569 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:49.128574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:49.128578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:49.129059 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:49.129328 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:49.130016 (+ 688us) server_negotiation.cc:299] Negotiation successful
0927 15:16:49.130293 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:16:49.131960 26387 raft_consensus.cc:2258] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:49.132401 26073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:49.132505 25874 catalog_manager.cc:4841] TS ccece2bca11944899ae41baa61d49801 (127.10.33.66:35461): tablet e0665b6bce174441bc5c6116daa3112f (table test-table [id=2ca8e9d6230a482686000a0a3e7c9396]) successfully deleted
I20260927 15:16:49.133181 26388 ts_tablet_manager.cc:1918] T e0665b6bce174441bc5c6116daa3112f P 058c7d5d33ad46af8d87df79504209c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:16:49.133358 26388 log.cc:1196] T e0665b6bce174441bc5c6116daa3112f P 058c7d5d33ad46af8d87df79504209c5: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/e0665b6bce174441bc5c6116daa3112f
I20260927 15:16:49.133913 26388 ts_tablet_manager.cc:1939] T e0665b6bce174441bc5c6116daa3112f P 058c7d5d33ad46af8d87df79504209c5: Deleting consensus metadata
I20260927 15:16:49.135735 25874 catalog_manager.cc:4841] TS 058c7d5d33ad46af8d87df79504209c5 (127.10.33.67:32889): tablet e0665b6bce174441bc5c6116daa3112f (table test-table [id=2ca8e9d6230a482686000a0a3e7c9396]) successfully deleted
I20260927 15:16:49.232172 26387 ts_tablet_manager.cc:1905] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:49.234990 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 25950
I20260927 15:16:49.249881 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 26084
W20260927 15:16:49.250089 25876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.33.65:33097 (error 108)
I20260927 15:16:49.252014 26359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:49.251538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33097 (local address 127.0.0.1:49844)
0927 15:16:49.251714 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:49.251840 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 15:16:49.252293 25876 catalog_manager.cc:4584] TS 64e324de2d014f7980bf068d1cf49792 (127.10.33.65:33097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0665b6bce174441bc5c6116daa3112f: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33097: connect: Connection refused (error 111)
Sep 27 15:16:49 dist-test-slave-f5h6 krb5kdc[25844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.66@KRBTEST.COM
I20260927 15:16:49.265244 26362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:49.250724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50326 (local address 127.10.33.66:35461)
0927 15:16:49.251020 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:49.251026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:49.251049 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:16:49.251277 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:49.251281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:49.251355 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:49.251455 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:49.253049 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.253811 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.255317 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.255789 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.255922 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:49.259919 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:49.259934 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:49.259948 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:49.259994 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:49.262691 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:49.263319 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:49.263327 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:49.263329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:49.263472 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:49.263782 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:49.263787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:49.263789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:49.264229 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:49.264510 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:49.264722 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 15:16:49.264889 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 15:16:49.265929 26207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:49.306005 26359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:49.305439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33097 (local address 127.0.0.1:49850)
0927 15:16:49.305641 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:49.305746 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 15:16:49.307237 26359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:49.306750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33097 (local address 127.0.0.1:49858)
0927 15:16:49.306984 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:49.307088 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 15:16:49.354904 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 26084
I20260927 15:16:49.367890 26359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:49.367402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33097 (local address 127.0.0.1:49868)
0927 15:16:49.367644 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:49.367731 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 15:16:49.368678 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 26217
I20260927 15:16:49.369146 26359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:49.368782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33097 (local address 127.0.0.1:49876)
0927 15:16:49.368928 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:49.369015 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
Sep 27 15:16:49 dist-test-slave-f5h6 krb5kdc[25844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.67@KRBTEST.COM
I20260927 15:16:49.383392 26358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:49.369341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57842 (local address 127.10.33.67:32889)
0927 15:16:49.369499 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:49.369504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:49.369643 (+ 139us) server_negotiation.cc:407] Connection header received
0927 15:16:49.369772 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:49.369776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:49.369848 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:49.370060 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:49.371346 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.372100 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.373382 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.373788 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.373934 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:49.378233 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:49.378250 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:49.378266 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:49.378301 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:49.380983 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:49.381596 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:49.381602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:49.381605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:49.381728 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:49.382081 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:49.382084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:49.382086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:49.382421 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:49.382644 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:49.382972 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 15:16:49.383139 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:16:49.384033 26340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:49.449914 26359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:49.449210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33097 (local address 127.0.0.1:49892)
0927 15:16:49.449518 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:49.449646 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 15:16:49.451166 26359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:49.450706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33097 (local address 127.0.0.1:49904)
0927 15:16:49.450920 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:49.451019 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260927 15:16:49.473539 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 26217
I20260927 15:16:49.487555 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 25857
I20260927 15:16:49.488920 25919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:16:49.600052 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 25857
I20260927 15:17:09.618162 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:39747
--webserver_interface=127.10.33.126
--webserver_port=42369
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:39747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:10.023334 26396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:10.023716 26396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:10.023818 26396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:10.032418 26396 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 15:17:10.032577 26396 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 15:17:10.032619 26396 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 15:17:10.032678 26396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:17:10.032738 26396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:10.032817 26396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:10.032867 26396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:17:10.032924 26396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:17:10.044371 26396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:39747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:39747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=42369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:17:10.046186 26396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:10.047798 26396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:10.075011 26403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:10.075047 26402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:10.075450 26396 server_base.cc:1034] running on GCE node
W20260927 15:17:10.075084 26405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:10.076912 26396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:10.078919 26396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:10.080183 26396 hybrid_clock.cc:648] HybridClock initialized: now 1790522230080092 us; error 57 us; skew 500 ppm
I20260927 15:17:10.080719 26396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:17:10 dist-test-slave-f5h6 krb5kdc[25844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522230, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:10.084684 26396 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:17:10.086794 26396 webserver.cc:460] Webserver started at http://127.10.33.126:42369/ using document root <none> and password file <none>
I20260927 15:17:10.087685 26396 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:10.087797 26396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:10.095322 26396 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:17:10.099087 26412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:10.101028 26396 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:17:10.101289 26396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "23b79be44f5247afa40b92844cbc226b"
format_stamp: "Formatted at 2026-09-27 15:16:47 on dist-test-slave-f5h6"
server_key: "d25cf3b488396fc22d1a9528f65f5641"
server_key_iv: "e0892c4c47db89621c56fffa0cf1e2b4"
server_key_version: "encryptionkey@0"
I20260927 15:17:10.102579 26396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:10.116757 26396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:10.121433 26396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:10.121729 26396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:10.142261 26396 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:39747
I20260927 15:17:10.142310 26463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:39747 every 8 connection(s)
I20260927 15:17:10.144008 26396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:17:10.146394 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 26396
I20260927 15:17:10.147564 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:33097
--local_ip_for_outbound_sockets=127.10.33.65
--tserver_master_addrs=127.10.33.126:39747
--webserver_port=45933
--webserver_interface=127.10.33.65
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:17:10.155541 26464 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:17:10.160925 26464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b: Bootstrap starting.
I20260927 15:17:10.186334 26464 log.cc:824] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:10.201372 26464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:17:10.202057 26464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b: Bootstrap complete.
I20260927 15:17:10.212759 26464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b79be44f5247afa40b92844cbc226b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39747 } }
I20260927 15:17:10.213820 26464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b79be44f5247afa40b92844cbc226b, State: Initialized, Role: FOLLOWER
I20260927 15:17:10.214633 26464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b79be44f5247afa40b92844cbc226b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39747 } }
I20260927 15:17:10.214902 26464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:17:10.215035 26464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:17:10.215209 26464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:17:10.219929 26464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b79be44f5247afa40b92844cbc226b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39747 } }
I20260927 15:17:10.220615 26464 leader_election.cc:302] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23b79be44f5247afa40b92844cbc226b; no voters:
I20260927 15:17:10.221343 26464 leader_election.cc:288] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:17:10.221508 26468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:17:10.222803 26468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 2 LEADER]: Becoming Leader. State: Replica: 23b79be44f5247afa40b92844cbc226b, State: Running, Role: LEADER
I20260927 15:17:10.223541 26468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b79be44f5247afa40b92844cbc226b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39747 } }
I20260927 15:17:10.224174 26464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:17:10.226580 26470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23b79be44f5247afa40b92844cbc226b. Latest consensus state: current_term: 2 leader_uuid: "23b79be44f5247afa40b92844cbc226b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b79be44f5247afa40b92844cbc226b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39747 } } }
I20260927 15:17:10.227131 26470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:10.227676 26469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "23b79be44f5247afa40b92844cbc226b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b79be44f5247afa40b92844cbc226b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39747 } } }
I20260927 15:17:10.227994 26469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:10.239153 26473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:17:10.249109 26473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:17:10.250967 26473 catalog_manager.cc:1259] Loaded cluster ID: eb15d132df1841e187802297a62c16de
I20260927 15:17:10.251163 26473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:17:10.255882 26473 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:17:10.257894 26473 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b: Loaded TSK: 0
I20260927 15:17:10.292668 26473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b: Generated new TSK 1
I20260927 15:17:10.293233 26473 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:17:10.520709 26466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:10.521001 26466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:10.521130 26466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:10.528532 26466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:10.528654 26466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:10.528806 26466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:17:10.538622 26466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:33097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=45933
--tserver_master_addrs=127.10.33.126:39747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:17:10.540051 26466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:10.541379 26466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:10.555716 26491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:10.555766 26494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:10.556499 26466 server_base.cc:1034] running on GCE node
W20260927 15:17:10.558305 26492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:10.559149 26466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:10.560256 26466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:10.561486 26466 hybrid_clock.cc:648] HybridClock initialized: now 1790522230561431 us; error 49 us; skew 500 ppm
I20260927 15:17:10.561801 26466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:17:10 dist-test-slave-f5h6 krb5kdc[25844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522230, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:10.564919 26466 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:17:10.567173 26466 webserver.cc:460] Webserver started at http://127.10.33.65:45933/ using document root <none> and password file <none>
I20260927 15:17:10.567911 26466 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:10.568051 26466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:10.573731 26466 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 15:17:10.576924 26501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:10.578471 26466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:17:10.578644 26466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "64e324de2d014f7980bf068d1cf49792"
format_stamp: "Formatted at 2026-09-27 15:16:47 on dist-test-slave-f5h6"
server_key: "c364214df20d066e5ebf77b0b1d766e8"
server_key_iv: "c464ba63cefedcd00bc822a8213c42f9"
server_key_version: "encryptionkey@0"
I20260927 15:17:10.579669 26466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:10.604333 26466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:10.609146 26466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:10.609438 26466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:10.610626 26466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:10.612848 26508 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260927 15:17:10.618639 26466 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 15:17:10.618758 26466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260927 15:17:10.618870 26466 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 15:17:10.622988 26466 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 15:17:10.623076 26466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20260927 15:17:10.623558 26508 tablet_bootstrap.cc:492] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792: Bootstrap starting.
I20260927 15:17:10.632055 26508 log.cc:824] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:10.645025 26508 tablet_bootstrap.cc:492] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:17:10.645771 26508 tablet_bootstrap.cc:492] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792: Bootstrap complete.
I20260927 15:17:10.646322 26508 ts_tablet_manager.cc:1397] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792: Time spent bootstrapping tablet: real 0.023s user 0.022s sys 0.000s
I20260927 15:17:10.656600 26508 raft_consensus.cc:348] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 } } peers { permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 } } peers { permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 } }
I20260927 15:17:10.658154 26508 raft_consensus.cc:729] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64e324de2d014f7980bf068d1cf49792, State: Initialized, Role: FOLLOWER
I20260927 15:17:10.659046 26508 consensus_queue.cc:260] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 } } peers { permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 } } peers { permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 } }
I20260927 15:17:10.660526 26508 ts_tablet_manager.cc:1428] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260927 15:17:10.666545 26466 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:33097
I20260927 15:17:10.666606 26616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:33097 every 8 connection(s)
I20260927 15:17:10.667732 26466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:17:10.669607 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 26466
I20260927 15:17:10.670282 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:35461
--local_ip_for_outbound_sockets=127.10.33.66
--tserver_master_addrs=127.10.33.126:39747
--webserver_port=36465
--webserver_interface=127.10.33.66
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:17:10 dist-test-slave-f5h6 krb5kdc[25844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522230, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:17:10.692847 26620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:10.670972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:59537 (local address 127.10.33.126:39747)
0927 15:17:10.671664 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:10.671683 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:10.672463 (+ 780us) server_negotiation.cc:407] Connection header received
0927 15:17:10.673329 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:10.673347 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:10.673759 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:10.674189 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:10.675829 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:10.677577 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:10.679963 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:10.680726 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:10.680898 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:10.685069 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:10.685077 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:10.685094 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:10.685137 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:10.688050 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:10.689008 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:10.689016 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:10.689022 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:10.689179 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:10.689659 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:10.689663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:10.689665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:10.690222 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:10.690383 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:10.691365 (+ 982us) server_negotiation.cc:299] Negotiation successful
0927 15:17:10.691734 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":265,"threads_started":1}
I20260927 15:17:10.693054 26619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:10.670847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.65:59537)
0927 15:17:10.672262 (+ 1415us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:10.672305 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:10.672805 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:10.674357 (+ 1552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:10.674365 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:10.674979 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:10.675597 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:10.675623 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:10.677785 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:10.677789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:10.679786 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:10.679804 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:10.680942 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:10.680949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:10.681244 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:10.682009 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:10.682040 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:10.684794 (+ 2754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:10.688199 (+ 3405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:10.688207 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:10.688223 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:10.688853 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:10.689298 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:10.689303 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:10.689307 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:10.689536 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:10.690357 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:10.690366 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:10.690806 (+ 440us) client_negotiation.cc:769] Sending connection context
0927 15:17:10.691157 (+ 351us) client_negotiation.cc:240] Negotiation successful
0927 15:17:10.691737 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1122,"thread_start_us":922,"threads_started":1}
I20260927 15:17:10.695883 26617 heartbeater.cc:347] Connected to a master server at 127.10.33.126:39747
I20260927 15:17:10.696190 26617 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:10.696872 26617 heartbeater.cc:511] Master 127.10.33.126:39747 requested a full tablet report, sending...
I20260927 15:17:10.699941 26429 ts_manager.cc:194] Registered new tserver with Master: 64e324de2d014f7980bf068d1cf49792 (127.10.33.65:33097)
I20260927 15:17:10.704375 26429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:59537
I20260927 15:17:10.714138 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:10.703652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36368 (local address 127.10.33.65:33097)
0927 15:17:10.706762 (+ 3110us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:10.706775 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:10.707155 (+ 380us) server_negotiation.cc:407] Connection header received
0927 15:17:10.707476 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:10.707485 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:10.707725 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:10.708054 (+ 329us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:10.708738 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:10.709746 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:10.712173 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:10.713065 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:10.713177 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:10.713335 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:10.713674 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 15:17:10.713786 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2881,"thread_start_us":2719,"threads_started":1}
I20260927 15:17:10.714201 26622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:10.703486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33097 (local address 127.0.0.1:36368)
0927 15:17:10.706979 (+ 3493us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:10.707021 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:10.707239 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:10.708069 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:10.708076 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:10.708107 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:10.708550 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:10.708562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:10.709914 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:10.709920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:10.712007 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:10.712023 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:10.713282 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:10.713288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:10.713467 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:10.713514 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 15:17:10.713642 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 15:17:10.713819 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3276,"thread_start_us":3951,"threads_started":1}
I20260927 15:17:10.715515 26551 tablet_service.cc:1514] Processing DeleteTablet for tablet e0665b6bce174441bc5c6116daa3112f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:49 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:36368
I20260927 15:17:10.718056 26624 tablet_replica.cc:330] stopping tablet replica
I20260927 15:17:10.718443 26624 raft_consensus.cc:2229] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:17:10.718798 26624 raft_consensus.cc:2258] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:10.719966 26624 ts_tablet_manager.cc:1905] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:10.726802 26624 ts_tablet_manager.cc:1918] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:10.726944 26624 log.cc:1196] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/e0665b6bce174441bc5c6116daa3112f
I20260927 15:17:10.727332 26624 ts_tablet_manager.cc:1939] T e0665b6bce174441bc5c6116daa3112f P 64e324de2d014f7980bf068d1cf49792: Deleting consensus metadata
I20260927 15:17:10.729527 26416 catalog_manager.cc:4841] TS 64e324de2d014f7980bf068d1cf49792 (127.10.33.65:33097): tablet e0665b6bce174441bc5c6116daa3112f (table test-table [id=2ca8e9d6230a482686000a0a3e7c9396]) successfully deleted
W20260927 15:17:11.011492 26621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:11.011812 26621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:11.011940 26621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:11.019280 26621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:11.019429 26621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:11.019584 26621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:17:11.030802 26621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:35461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=36465
--tserver_master_addrs=127.10.33.126:39747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:17:11.032219 26621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:11.033646 26621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:11.046837 26630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:11.046895 26632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:11.046813 26629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:11.048138 26621 server_base.cc:1034] running on GCE node
I20260927 15:17:11.048799 26621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:11.049825 26621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:11.051074 26621 hybrid_clock.cc:648] HybridClock initialized: now 1790522231051027 us; error 41 us; skew 500 ppm
I20260927 15:17:11.051403 26621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:17:11 dist-test-slave-f5h6 krb5kdc[25844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522231, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:11.054550 26621 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:17:11.056090 26621 webserver.cc:460] Webserver started at http://127.10.33.66:36465/ using document root <none> and password file <none>
I20260927 15:17:11.056747 26621 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:11.056861 26621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:11.062506 26621 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260927 15:17:11.065516 26639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:11.067227 26621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:17:11.067408 26621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "ccece2bca11944899ae41baa61d49801"
format_stamp: "Formatted at 2026-09-27 15:16:48 on dist-test-slave-f5h6"
server_key: "d0920b366530a4e93fbfeb89a670ceee"
server_key_iv: "2e7bc8d665cd29f159ef31ca959ca4fb"
server_key_version: "encryptionkey@0"
I20260927 15:17:11.068468 26621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:11.111697 26621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:11.115949 26621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:11.116185 26621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:11.117172 26621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:11.118561 26621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:11.118646 26621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:11.118734 26621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:11.118786 26621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:11.150097 26621 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:35461
I20260927 15:17:11.150144 26751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:35461 every 8 connection(s)
I20260927 15:17:11.151973 26621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:17:11.157562 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 26621
I20260927 15:17:11.158463 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:32889
--local_ip_for_outbound_sockets=127.10.33.67
--tserver_master_addrs=127.10.33.126:39747
--webserver_port=33701
--webserver_interface=127.10.33.67
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:17:11 dist-test-slave-f5h6 krb5kdc[25844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522231, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:17:11.176167 26620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.155025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:37469 (local address 127.10.33.126:39747)
0927 15:17:11.155196 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.155200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.156762 (+ 1562us) server_negotiation.cc:407] Connection header received
0927 15:17:11.159667 (+ 2905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.159674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.159795 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.159911 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:11.162152 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.162954 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.165396 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.165891 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.166030 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.170041 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:11.170048 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:11.170053 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:11.170095 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:11.172899 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.173735 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.173739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.173742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.173918 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.174354 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.174359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.174362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.174648 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:11.174789 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.175532 (+ 743us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.175808 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:17:11.177161 26754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.155571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.66:37469)
0927 15:17:11.156577 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:11.156621 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:11.159265 (+ 2644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:11.160233 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:11.160249 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:11.161033 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:11.161924 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.161957 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.163205 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.163212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.165179 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.165199 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.166149 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.166158 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.166465 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.167375 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:11.167402 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:11.169712 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:11.173059 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:11.173067 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:11.173081 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:11.173600 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:11.174024 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:11.174028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:11.174030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:11.174241 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:11.174794 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:11.174803 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:11.175166 (+ 363us) client_negotiation.cc:769] Sending connection context
0927 15:17:11.175428 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 15:17:11.175994 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":482,"threads_started":1}
I20260927 15:17:11.178915 26752 heartbeater.cc:347] Connected to a master server at 127.10.33.126:39747
I20260927 15:17:11.179275 26752 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:11.180006 26752 heartbeater.cc:511] Master 127.10.33.126:39747 requested a full tablet report, sending...
I20260927 15:17:11.181586 26429 ts_manager.cc:194] Registered new tserver with Master: ccece2bca11944899ae41baa61d49801 (127.10.33.66:35461)
I20260927 15:17:11.182449 26429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:37469
W20260927 15:17:11.492074 26755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:11.492393 26755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:11.492528 26755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:11.499835 26755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:11.499967 26755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:11.500113 26755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:17:11.510093 26755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:32889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=33701
--tserver_master_addrs=127.10.33.126:39747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:17:11.511400 26755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:11.512693 26755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:11.526319 26760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:11.526268 26763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:11.526343 26761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:11.527257 26755 server_base.cc:1034] running on GCE node
I20260927 15:17:11.527884 26755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:11.528900 26755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:11.530107 26755 hybrid_clock.cc:648] HybridClock initialized: now 1790522231530028 us; error 66 us; skew 500 ppm
I20260927 15:17:11.530429 26755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:17:11 dist-test-slave-f5h6 krb5kdc[25844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522231, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:11.533720 26755 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:17:11.535425 26755 webserver.cc:460] Webserver started at http://127.10.33.67:33701/ using document root <none> and password file <none>
I20260927 15:17:11.536100 26755 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:11.536217 26755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:11.541661 26755 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:17:11.544425 26770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:11.545830 26755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:17:11.546039 26755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "058c7d5d33ad46af8d87df79504209c5"
format_stamp: "Formatted at 2026-09-27 15:16:48 on dist-test-slave-f5h6"
server_key: "e29ea2a1e4c62e9961ad206000a5d107"
server_key_iv: "5cc9d537c274292370bc0518228972ac"
server_key_version: "encryptionkey@0"
I20260927 15:17:11.547005 26755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:11.565501 26755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:11.569710 26755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:11.570001 26755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:11.570963 26755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:11.572309 26755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:11.572404 26755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:11.572484 26755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:11.572540 26755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:11.603683 26755 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:32889
I20260927 15:17:11.603785 26882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:32889 every 8 connection(s)
I20260927 15:17:11.605264 26755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:17:11.615751 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 26755
Sep 27 15:17:11 dist-test-slave-f5h6 krb5kdc[25844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522231, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:17:11.625538 26620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.607922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:46513 (local address 127.10.33.126:39747)
0927 15:17:11.608089 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.608093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.609343 (+ 1250us) server_negotiation.cc:407] Connection header received
0927 15:17:11.611245 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.611251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.611333 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.611420 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:11.612926 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.613632 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.615226 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.615829 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.615960 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.619775 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:11.619780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:11.619783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:11.619815 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:11.622444 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.623181 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.623185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.623188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.623318 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.623740 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.623743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.623745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.624006 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:11.624148 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.624901 (+ 753us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.625165 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:17:11.626657 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.608383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.67:46513)
0927 15:17:11.609122 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:11.609160 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:11.610956 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:11.611673 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:11.611681 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:11.612198 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:11.612760 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.612777 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.613800 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.613804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.615084 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.615097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.616084 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.616090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.616312 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.616937 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:11.616972 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:11.619545 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:11.622581 (+ 3036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:11.622591 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:11.622610 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:11.623057 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:11.623421 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:11.623426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:11.623431 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:11.623626 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:11.624130 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:11.624139 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:11.624497 (+ 358us) client_negotiation.cc:769] Sending connection context
0927 15:17:11.624803 (+ 306us) client_negotiation.cc:240] Negotiation successful
0927 15:17:11.625347 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":297,"threads_started":1}
I20260927 15:17:11.628190 26883 heartbeater.cc:347] Connected to a master server at 127.10.33.126:39747
I20260927 15:17:11.628532 26883 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:11.629420 26883 heartbeater.cc:511] Master 127.10.33.126:39747 requested a full tablet report, sending...
I20260927 15:17:11.630945 26886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.616706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40748 (local address 127.10.33.126:39747)
0927 15:17:11.617217 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.617224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.617675 (+ 451us) server_negotiation.cc:407] Connection header received
0927 15:17:11.617837 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.617840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.617934 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.618021 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:11.619563 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.620443 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.621826 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.622313 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.622420 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.624933 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:11.624938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:11.624941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:11.624969 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:11.627915 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.628821 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.628827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.628829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.628952 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.629409 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.629413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.629415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.629700 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:11.629962 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.630344 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.630579 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":268,"threads_started":1}
I20260927 15:17:11.631132 26429 ts_manager.cc:194] Registered new tserver with Master: 058c7d5d33ad46af8d87df79504209c5 (127.10.33.67:32889)
I20260927 15:17:11.632217 26429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:46513
I20260927 15:17:11.633672 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:17:11.640754 26886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.634674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40756 (local address 127.10.33.126:39747)
0927 15:17:11.634853 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.634859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.635310 (+ 451us) server_negotiation.cc:407] Connection header received
0927 15:17:11.635547 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.635551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.635685 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.635812 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:17:11.636222 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.636921 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.638618 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.639124 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.639282 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.639825 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:17:11.640075 (+ 250us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 15:17:11.640526 (+ 451us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40756: token expired
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 15:17:11.640867 26886 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40756: token expired
W20260927 15:17:11.641465 26351 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.10.33.126:39747)
I20260927 15:17:11.655054 26886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.641997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40768 (local address 127.10.33.126:39747)
0927 15:17:11.642231 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.642238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.642261 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:17:11.642322 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.642327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.642408 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.642513 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:11.643902 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.644691 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.646216 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.646666 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.646772 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.649834 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:11.649841 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:11.649844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:11.649920 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:11.652490 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.653164 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.653168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.653170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.653338 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.653660 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.653662 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.653664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.654054 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:11.654240 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.654632 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.654826 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:17:11.659217 26428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:17:11.666858 26428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:17:11.679908 26551 tablet_service.cc:1467] Processing CreateTablet for tablet 592934b040f448f88fa4ff473b4690b7 (DEFAULT_TABLE table=test-table [id=62b580b51e1f46c2805229696a4f12f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:11.680882 26551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 592934b040f448f88fa4ff473b4690b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:11.689729 26894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.678744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39180 (local address 127.10.33.67:32889)
0927 15:17:11.679399 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.679405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.680046 (+ 641us) server_negotiation.cc:407] Connection header received
0927 15:17:11.680389 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.680397 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.680690 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.680888 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:11.681926 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.683663 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.686443 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.688069 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.688182 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.688444 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.689154 (+ 710us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.689357 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":353,"threads_started":1}
I20260927 15:17:11.690512 26893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.678473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:32889 (local address 127.0.0.1:39180)
0927 15:17:11.679924 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:11.679940 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:11.680164 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:11.681228 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:11.681234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:11.681261 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:11.681655 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.681667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.683904 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.683910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.686231 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.686251 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.688706 (+ 2455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.688714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.688925 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.688962 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 15:17:11.689987 (+ 1025us) client_negotiation.cc:240] Negotiation successful
0927 15:17:11.690163 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1348,"thread_start_us":1199,"threads_started":1}
I20260927 15:17:11.691459 26897 tablet_bootstrap.cc:492] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792: Bootstrap starting.
I20260927 15:17:11.693347 26897 tablet_bootstrap.cc:654] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:11.695487 26818 tablet_service.cc:1467] Processing CreateTablet for tablet 592934b040f448f88fa4ff473b4690b7 (DEFAULT_TABLE table=test-table [id=62b580b51e1f46c2805229696a4f12f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:11.697587 26818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 592934b040f448f88fa4ff473b4690b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:11.699393 26897 tablet_bootstrap.cc:492] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792: No bootstrap required, opened a new log
I20260927 15:17:11.699544 26897 ts_tablet_manager.cc:1397] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260927 15:17:11.700292 26897 raft_consensus.cc:348] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 } } peers { permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 } } peers { permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 } }
I20260927 15:17:11.700598 26897 raft_consensus.cc:374] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:11.701011 26897 raft_consensus.cc:729] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64e324de2d014f7980bf068d1cf49792, State: Initialized, Role: FOLLOWER
I20260927 15:17:11.701083 26896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.680321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48222 (local address 127.10.33.66:35461)
0927 15:17:11.691762 (+ 11441us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.691783 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.691815 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:17:11.691934 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.691942 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.692234 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.693092 (+ 858us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:11.693309 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.694940 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.698535 (+ 3595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.699932 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.700078 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.700266 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.700575 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.700760 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11270,"thread_start_us":17761,"threads_started":1}
I20260927 15:17:11.701293 26897 consensus_queue.cc:260] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 } } peers { permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 } } peers { permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 } }
I20260927 15:17:11.702152 26897 ts_tablet_manager.cc:1428] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260927 15:17:11.702209 26895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.680062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:35461 (local address 127.0.0.1:48222)
0927 15:17:11.689785 (+ 9723us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:11.689804 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:11.689956 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:11.692470 (+ 2514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:11.692475 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:11.692507 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:11.692878 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.692891 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.695113 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.695119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.698328 (+ 3209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.698345 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.700159 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.700166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.700350 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.700384 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 15:17:11.701701 (+ 1317us) client_negotiation.cc:240] Negotiation successful
0927 15:17:11.701881 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9555,"thread_start_us":9971,"threads_started":1}
I20260927 15:17:11.702176 26617 heartbeater.cc:503] Master 127.10.33.126:39747 was elected leader, sending a full tablet report...
I20260927 15:17:11.706036 26687 tablet_service.cc:1467] Processing CreateTablet for tablet 592934b040f448f88fa4ff473b4690b7 (DEFAULT_TABLE table=test-table [id=62b580b51e1f46c2805229696a4f12f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:11.707477 26687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 592934b040f448f88fa4ff473b4690b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:11.711956 26900 tablet_bootstrap.cc:492] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5: Bootstrap starting.
I20260927 15:17:11.715906 26900 tablet_bootstrap.cc:654] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:11.717267 26900 log.cc:824] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:11.720422 26900 tablet_bootstrap.cc:492] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5: No bootstrap required, opened a new log
I20260927 15:17:11.720738 26900 ts_tablet_manager.cc:1397] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260927 15:17:11.722414 26902 tablet_bootstrap.cc:492] T 592934b040f448f88fa4ff473b4690b7 P ccece2bca11944899ae41baa61d49801: Bootstrap starting.
I20260927 15:17:11.725908 26902 tablet_bootstrap.cc:654] T 592934b040f448f88fa4ff473b4690b7 P ccece2bca11944899ae41baa61d49801: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:11.727341 26902 log.cc:824] T 592934b040f448f88fa4ff473b4690b7 P ccece2bca11944899ae41baa61d49801: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:11.729239 26900 raft_consensus.cc:348] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 } } peers { permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 } } peers { permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 } }
I20260927 15:17:11.729620 26900 raft_consensus.cc:374] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:11.729720 26900 raft_consensus.cc:729] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 058c7d5d33ad46af8d87df79504209c5, State: Initialized, Role: FOLLOWER
I20260927 15:17:11.730299 26900 consensus_queue.cc:260] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 } } peers { permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 } } peers { permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 } }
I20260927 15:17:11.730660 26902 tablet_bootstrap.cc:492] T 592934b040f448f88fa4ff473b4690b7 P ccece2bca11944899ae41baa61d49801: No bootstrap required, opened a new log
I20260927 15:17:11.730940 26902 ts_tablet_manager.cc:1397] T 592934b040f448f88fa4ff473b4690b7 P ccece2bca11944899ae41baa61d49801: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260927 15:17:11.731395 26883 heartbeater.cc:503] Master 127.10.33.126:39747 was elected leader, sending a full tablet report...
I20260927 15:17:11.731855 26900 ts_tablet_manager.cc:1428] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260927 15:17:11.739671 26902 raft_consensus.cc:348] T 592934b040f448f88fa4ff473b4690b7 P ccece2bca11944899ae41baa61d49801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 } } peers { permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 } } peers { permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 } }
I20260927 15:17:11.740067 26902 raft_consensus.cc:374] T 592934b040f448f88fa4ff473b4690b7 P ccece2bca11944899ae41baa61d49801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:11.740139 26902 raft_consensus.cc:729] T 592934b040f448f88fa4ff473b4690b7 P ccece2bca11944899ae41baa61d49801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccece2bca11944899ae41baa61d49801, State: Initialized, Role: FOLLOWER
I20260927 15:17:11.740674 26902 consensus_queue.cc:260] T 592934b040f448f88fa4ff473b4690b7 P ccece2bca11944899ae41baa61d49801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 } } peers { permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 } } peers { permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 } }
I20260927 15:17:11.741744 26752 heartbeater.cc:503] Master 127.10.33.126:39747 was elected leader, sending a full tablet report...
I20260927 15:17:11.742375 26902 ts_tablet_manager.cc:1428] T 592934b040f448f88fa4ff473b4690b7 P ccece2bca11944899ae41baa61d49801: Time spent starting tablet: real 0.011s user 0.009s sys 0.003s
I20260927 15:17:11.796540 26904 raft_consensus.cc:482] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:17:11.796849 26904 raft_consensus.cc:504] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 } } peers { permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 } } peers { permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 } }
I20260927 15:17:11.798756 26904 leader_election.cc:288] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccece2bca11944899ae41baa61d49801 (127.10.33.66:35461), 64e324de2d014f7980bf068d1cf49792 (127.10.33.65:33097)
I20260927 15:17:11.807004 26896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.799441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:35619 (local address 127.10.33.66:35461)
0927 15:17:11.799654 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.799661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.800145 (+ 484us) server_negotiation.cc:407] Connection header received
0927 15:17:11.800247 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.800252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.800333 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.800421 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:11.801016 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.802098 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.804522 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.805807 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.806007 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.806094 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.806550 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.806702 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 15:17:11.806998 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.799067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33097 (local address 127.10.33.67:32919)
0927 15:17:11.799363 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:11.799386 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:11.799517 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:11.800360 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:11.800364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:11.800402 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:11.800621 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.800636 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.802078 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.802082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.804309 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.804327 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.806035 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.806042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.806295 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.806352 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 15:17:11.806469 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 15:17:11.806655 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":107}
I20260927 15:17:11.807045 26906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.799238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:32919 (local address 127.10.33.65:33097)
0927 15:17:11.800038 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.800042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.800057 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:17:11.800119 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.800123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.800195 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.800303 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:11.800772 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.801919 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.804522 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.805801 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.806023 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.806094 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.806550 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.806712 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":585,"threads_started":1}
I20260927 15:17:11.807091 26907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.799188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:35461 (local address 127.10.33.67:35619)
0927 15:17:11.800041 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:11.800054 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:11.800155 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:11.800467 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:11.800471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:11.800490 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:11.800876 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.800886 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.802232 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.802238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.804309 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.804327 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.806069 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.806075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.806288 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.806357 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 15:17:11.806468 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 15:17:11.806723 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":94,"thread_start_us":565,"threads_started":1}
I20260927 15:17:11.808239 26707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592934b040f448f88fa4ff473b4690b7" candidate_uuid: "058c7d5d33ad46af8d87df79504209c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccece2bca11944899ae41baa61d49801" is_pre_election: true
I20260927 15:17:11.808264 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592934b040f448f88fa4ff473b4690b7" candidate_uuid: "058c7d5d33ad46af8d87df79504209c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64e324de2d014f7980bf068d1cf49792" is_pre_election: true
I20260927 15:17:11.808724 26707 raft_consensus.cc:2454] T 592934b040f448f88fa4ff473b4690b7 P ccece2bca11944899ae41baa61d49801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 058c7d5d33ad46af8d87df79504209c5 in term 0.
I20260927 15:17:11.808724 26571 raft_consensus.cc:2454] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 058c7d5d33ad46af8d87df79504209c5 in term 0.
I20260927 15:17:11.809434 26772 leader_election.cc:302] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 058c7d5d33ad46af8d87df79504209c5, 64e324de2d014f7980bf068d1cf49792, ccece2bca11944899ae41baa61d49801; no voters:
I20260927 15:17:11.809841 26904 raft_consensus.cc:2781] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:17:11.810014 26904 raft_consensus.cc:482] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:17:11.810115 26904 raft_consensus.cc:3037] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:11.812285 26904 raft_consensus.cc:504] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 } } peers { permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 } } peers { permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 } }
I20260927 15:17:11.812938 26904 leader_election.cc:288] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5 [CANDIDATE]: Term 1 election: Requested vote from peers ccece2bca11944899ae41baa61d49801 (127.10.33.66:35461), 64e324de2d014f7980bf068d1cf49792 (127.10.33.65:33097)
I20260927 15:17:11.813436 26707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592934b040f448f88fa4ff473b4690b7" candidate_uuid: "058c7d5d33ad46af8d87df79504209c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccece2bca11944899ae41baa61d49801"
I20260927 15:17:11.813483 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592934b040f448f88fa4ff473b4690b7" candidate_uuid: "058c7d5d33ad46af8d87df79504209c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64e324de2d014f7980bf068d1cf49792"
I20260927 15:17:11.813630 26707 raft_consensus.cc:3037] T 592934b040f448f88fa4ff473b4690b7 P ccece2bca11944899ae41baa61d49801 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:11.813657 26571 raft_consensus.cc:3037] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:11.815933 26707 raft_consensus.cc:2454] T 592934b040f448f88fa4ff473b4690b7 P ccece2bca11944899ae41baa61d49801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 058c7d5d33ad46af8d87df79504209c5 in term 1.
I20260927 15:17:11.815973 26571 raft_consensus.cc:2454] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 058c7d5d33ad46af8d87df79504209c5 in term 1.
I20260927 15:17:11.816370 26772 leader_election.cc:302] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058c7d5d33ad46af8d87df79504209c5, ccece2bca11944899ae41baa61d49801; no voters:
I20260927 15:17:11.816658 26904 raft_consensus.cc:2781] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:17:11.817075 26904 raft_consensus.cc:686] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5 [term 1 LEADER]: Becoming Leader. State: Replica: 058c7d5d33ad46af8d87df79504209c5, State: Running, Role: LEADER
I20260927 15:17:11.817521 26904 consensus_queue.cc:237] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 } } peers { permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 } } peers { permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 } }
I20260927 15:17:11.822724 26429 catalog_manager.cc:5495] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 058c7d5d33ad46af8d87df79504209c5 (127.10.33.67). New cstate: current_term: 1 leader_uuid: "058c7d5d33ad46af8d87df79504209c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058c7d5d33ad46af8d87df79504209c5" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 } health_report { overall_health: UNKNOWN } } }
W20260927 15:17:11.856882 26884 tablet.cc:2369] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:17:11.865010 26894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.859581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39190 (local address 127.10.33.67:32889)
0927 15:17:11.859827 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.859834 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.859902 (+ 68us) server_negotiation.cc:407] Connection header received
0927 15:17:11.860083 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.860087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.860164 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.860289 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:17:11.860654 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.861445 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.862934 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.863339 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.863438 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.863928 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:17:11.864421 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:17:11.864528 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.864665 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.864739 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:17:11.873379 26707 raft_consensus.cc:1261] T 592934b040f448f88fa4ff473b4690b7 P ccece2bca11944899ae41baa61d49801 [term 1 FOLLOWER]: Refusing update from remote peer 058c7d5d33ad46af8d87df79504209c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:17:11.873363 26571 raft_consensus.cc:1261] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792 [term 1 FOLLOWER]: Refusing update from remote peer 058c7d5d33ad46af8d87df79504209c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:17:11.874359 26908 consensus_queue.cc:1035] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64e324de2d014f7980bf068d1cf49792" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:17:11.874686 26904 consensus_queue.cc:1035] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccece2bca11944899ae41baa61d49801" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:17:11.884054 26911 mvcc.cc:205] Tried to move back new op lower bound from 7333979061742858240 to 7333979061528539136. Current Snapshot: MvccSnapshot[applied={T|T < 7333979061742858240}]
I20260927 15:17:11.885669 26912 mvcc.cc:205] Tried to move back new op lower bound from 7333979061742858240 to 7333979061528539136. Current Snapshot: MvccSnapshot[applied={T|T < 7333979061742858240}]
I20260927 15:17:11.887934 26913 mvcc.cc:205] Tried to move back new op lower bound from 7333979061742858240 to 7333979061528539136. Current Snapshot: MvccSnapshot[applied={T|T < 7333979061742858240}]
I20260927 15:17:11.898051 26429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:17:11.898314 26429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:17:11.901793 26429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b: Sending DeleteTablet for 3 replicas of tablet 592934b040f448f88fa4ff473b4690b7
I20260927 15:17:11.902880 26818 tablet_service.cc:1514] Processing DeleteTablet for tablet 592934b040f448f88fa4ff473b4690b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:11 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:39180
I20260927 15:17:11.903120 26687 tablet_service.cc:1514] Processing DeleteTablet for tablet 592934b040f448f88fa4ff473b4690b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:11 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:48222
I20260927 15:17:11.903644 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 26466
I20260927 15:17:11.903726 26551 tablet_service.cc:1514] Processing DeleteTablet for tablet 592934b040f448f88fa4ff473b4690b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:11 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:36368
I20260927 15:17:11.905426 26921 tablet_replica.cc:330] T 592934b040f448f88fa4ff473b4690b7 P ccece2bca11944899ae41baa61d49801: stopping tablet replica
I20260927 15:17:11.905825 26921 raft_consensus.cc:2229] T 592934b040f448f88fa4ff473b4690b7 P ccece2bca11944899ae41baa61d49801 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:17:11.906373 26921 raft_consensus.cc:2258] T 592934b040f448f88fa4ff473b4690b7 P ccece2bca11944899ae41baa61d49801 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:11.907178 26920 tablet_replica.cc:330] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5: stopping tablet replica
I20260927 15:17:11.907595 26920 raft_consensus.cc:2229] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:17:11.908368 26920 raft_consensus.cc:2258] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:11.909119 26922 tablet_replica.cc:330] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792: stopping tablet replica
I20260927 15:17:11.909341 26922 raft_consensus.cc:2229] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:17:11.909703 26922 raft_consensus.cc:2258] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:11.910356 26921 ts_tablet_manager.cc:1905] T 592934b040f448f88fa4ff473b4690b7 P ccece2bca11944899ae41baa61d49801: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:11.910584 26920 ts_tablet_manager.cc:1905] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:11.917351 26920 ts_tablet_manager.cc:1918] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:11.917335 26921 ts_tablet_manager.cc:1918] T 592934b040f448f88fa4ff473b4690b7 P ccece2bca11944899ae41baa61d49801: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:11.917524 26921 log.cc:1196] T 592934b040f448f88fa4ff473b4690b7 P ccece2bca11944899ae41baa61d49801: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/592934b040f448f88fa4ff473b4690b7
I20260927 15:17:11.917524 26920 log.cc:1196] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/592934b040f448f88fa4ff473b4690b7
I20260927 15:17:11.918075 26921 ts_tablet_manager.cc:1939] T 592934b040f448f88fa4ff473b4690b7 P ccece2bca11944899ae41baa61d49801: Deleting consensus metadata
I20260927 15:17:11.918093 26920 ts_tablet_manager.cc:1939] T 592934b040f448f88fa4ff473b4690b7 P 058c7d5d33ad46af8d87df79504209c5: Deleting consensus metadata
I20260927 15:17:11.919569 26922 ts_tablet_manager.cc:1905] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:11.920317 26414 catalog_manager.cc:4841] TS 058c7d5d33ad46af8d87df79504209c5 (127.10.33.67:32889): tablet 592934b040f448f88fa4ff473b4690b7 (table test-table [id=62b580b51e1f46c2805229696a4f12f5]) successfully deleted
I20260927 15:17:11.922964 26414 catalog_manager.cc:4841] TS ccece2bca11944899ae41baa61d49801 (127.10.33.66:35461): tablet 592934b040f448f88fa4ff473b4690b7 (table test-table [id=62b580b51e1f46c2805229696a4f12f5]) successfully deleted
I20260927 15:17:11.925186 26906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.904817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36380 (local address 127.10.33.65:33097)
0927 15:17:11.905006 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.905012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.905031 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:17:11.905087 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.905093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.905166 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.905256 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:11.907231 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.908288 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.910474 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.911107 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.911234 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.914759 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:11.914779 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:11.914809 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:11.914859 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:11.919034 (+ 4175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.921277 (+ 2243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.921290 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.921293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.921476 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.923257 (+ 1781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.923274 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.923278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.923796 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:11.924112 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.924529 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.924806 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 15:17:11.925998 26612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:17:12.018695 26922 ts_tablet_manager.cc:1918] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:12.018932 26922 log.cc:1196] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/592934b040f448f88fa4ff473b4690b7
I20260927 15:17:12.019402 26922 ts_tablet_manager.cc:1939] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792: Deleting consensus metadata
I20260927 15:17:12.019418 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 26466
I20260927 15:17:12.033990 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 26621
W20260927 15:17:12.034121 26416 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.33.65:33097 (error 108)
I20260927 15:17:12.035827 26895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.035411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33097 (local address 127.0.0.1:36390)
0927 15:17:12.035569 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:12.035662 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 15:17:12.036217 26416 catalog_manager.cc:4584] TS 64e324de2d014f7980bf068d1cf49792 (127.10.33.65:33097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 592934b040f448f88fa4ff473b4690b7: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33097: connect: Connection refused (error 111)
I20260927 15:17:12.050328 26896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.034982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48226 (local address 127.10.33.66:35461)
0927 15:17:12.035208 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:12.035223 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:12.035474 (+ 251us) server_negotiation.cc:407] Connection header received
0927 15:17:12.035760 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:12.035774 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:12.035884 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:12.035991 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:12.037801 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.038815 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.040741 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.041155 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.041260 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.044602 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:12.044616 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:12.044633 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:12.044674 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:12.047389 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:12.048179 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:12.048185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:12.048188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:12.048306 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:12.048656 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:12.048659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:12.048661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:12.049029 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:12.049240 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:12.049715 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 15:17:12.049949 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 15:17:12.051060 26747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:17:12.089896 26895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.089325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33097 (local address 127.0.0.1:36406)
0927 15:17:12.089549 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:12.089668 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 15:17:12.091013 26895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.090604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33097 (local address 127.0.0.1:36416)
0927 15:17:12.090749 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:12.090843 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 15:17:12.140158 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 26621
I20260927 15:17:12.152079 26895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.151451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33097 (local address 127.0.0.1:36420)
0927 15:17:12.151754 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:12.151880 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260927 15:17:12.153501 26895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.153038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33097 (local address 127.0.0.1:36436)
0927 15:17:12.153265 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:12.153354 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 15:17:12.160967 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 26755
I20260927 15:17:12.178073 26894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.162090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39194 (local address 127.10.33.67:32889)
0927 15:17:12.162310 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:12.162316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:12.162336 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:17:12.162398 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:12.162403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:12.162490 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:12.162596 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:12.164299 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.165478 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.166997 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.167679 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.167819 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.170541 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:12.170556 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:12.170577 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:12.170631 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:12.174648 (+ 4017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:12.175537 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:12.175547 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:12.175550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:12.175765 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:12.176210 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:12.176226 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:12.176229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:12.176706 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:12.177030 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:12.177383 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 15:17:12.177619 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:17:12.178789 26878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:17:12.234462 26895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.233793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33097 (local address 127.0.0.1:36450)
0927 15:17:12.234068 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:12.234182 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260927 15:17:12.235560 26895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.235132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33097 (local address 127.0.0.1:36456)
0927 15:17:12.235346 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:12.235431 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 15:17:12.300053 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 26755
I20260927 15:17:12.317468 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 26396
I20260927 15:17:12.318586 26459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:17:12.434609 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 26396
I20260927 15:17:32.453058 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:39747
--webserver_interface=127.10.33.126
--webserver_port=42369
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:39747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:32.778007 26928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:32.778326 26928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:32.778417 26928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:32.785725 26928 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 15:17:32.785880 26928 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 15:17:32.785938 26928 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 15:17:32.785995 26928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:17:32.786042 26928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:32.786124 26928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:32.786178 26928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:17:32.786235 26928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:17:32.796458 26928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:39747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:39747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=42369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:17:32.797742 26928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:32.799093 26928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:32.812463 26933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:32.812564 26934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:32.812564 26936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:32.812862 26928 server_base.cc:1034] running on GCE node
I20260927 15:17:32.814100 26928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:32.815682 26928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:32.816978 26928 hybrid_clock.cc:648] HybridClock initialized: now 1790522252816897 us; error 52 us; skew 500 ppm
I20260927 15:17:32.817401 26928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:17:32 dist-test-slave-f5h6 krb5kdc[25844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522252, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:32.820901 26928 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:17:32.822628 26928 webserver.cc:460] Webserver started at http://127.10.33.126:42369/ using document root <none> and password file <none>
I20260927 15:17:32.823364 26928 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:32.823469 26928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:32.829452 26928 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:17:32.832976 26943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:32.834621 26928 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260927 15:17:32.834810 26928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "23b79be44f5247afa40b92844cbc226b"
format_stamp: "Formatted at 2026-09-27 15:16:47 on dist-test-slave-f5h6"
server_key: "d25cf3b488396fc22d1a9528f65f5641"
server_key_iv: "e0892c4c47db89621c56fffa0cf1e2b4"
server_key_version: "encryptionkey@0"
I20260927 15:17:32.835817 26928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:32.851982 26928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:32.856453 26928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:32.856760 26928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:32.876771 26928 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:39747
I20260927 15:17:32.876863 26994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:39747 every 8 connection(s)
I20260927 15:17:32.878401 26928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:17:32.879189 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 26928
I20260927 15:17:32.880093 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:33097
--local_ip_for_outbound_sockets=127.10.33.65
--tserver_master_addrs=127.10.33.126:39747
--webserver_port=45933
--webserver_interface=127.10.33.65
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:17:32.887511 26995 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:17:32.896231 26995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b: Bootstrap starting.
I20260927 15:17:32.923106 26995 log.cc:824] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:32.946055 26995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:17:32.946784 26995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b: Bootstrap complete.
I20260927 15:17:32.957366 26995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b79be44f5247afa40b92844cbc226b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39747 } }
I20260927 15:17:32.958536 26995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b79be44f5247afa40b92844cbc226b, State: Initialized, Role: FOLLOWER
I20260927 15:17:32.959313 26995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b79be44f5247afa40b92844cbc226b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39747 } }
I20260927 15:17:32.959569 26995 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:17:32.959692 26995 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:17:32.959862 26995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 2 FOLLOWER]: Advancing to term 3
I20260927 15:17:32.964499 26995 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b79be44f5247afa40b92844cbc226b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39747 } }
I20260927 15:17:32.965095 26995 leader_election.cc:302] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23b79be44f5247afa40b92844cbc226b; no voters:
I20260927 15:17:32.965819 26995 leader_election.cc:288] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 15:17:32.966017 26999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 3 FOLLOWER]: Leader election won for term 3
I20260927 15:17:32.967345 26999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [term 3 LEADER]: Becoming Leader. State: Replica: 23b79be44f5247afa40b92844cbc226b, State: Running, Role: LEADER
I20260927 15:17:32.968160 26999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b79be44f5247afa40b92844cbc226b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39747 } }
I20260927 15:17:32.968616 26995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:17:32.971061 27001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23b79be44f5247afa40b92844cbc226b. Latest consensus state: current_term: 3 leader_uuid: "23b79be44f5247afa40b92844cbc226b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b79be44f5247afa40b92844cbc226b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39747 } } }
I20260927 15:17:32.971515 27001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:32.972123 27000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "23b79be44f5247afa40b92844cbc226b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b79be44f5247afa40b92844cbc226b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39747 } } }
I20260927 15:17:32.972432 27000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:32.980082 27005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:17:32.991047 27005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:17:32.992647 27005 catalog_manager.cc:1259] Loaded cluster ID: eb15d132df1841e187802297a62c16de
I20260927 15:17:32.992805 27005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:17:32.997385 27005 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:17:32.999360 27005 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b: Loaded TSK: 0
I20260927 15:17:32.999600 27005 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23b79be44f5247afa40b92844cbc226b: Loaded TSK: 1
F20260927 15:17:33.000762 27005 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f14af56f9ad google::LogMessage::Fail() at ??:0
@ 0x7f14af57288a google::LogMessage::SendToLog() at ??:0
@ 0x7f14af56f4de google::LogMessage::Flush() at ??:0
@ 0x7f14af570a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f14b32d37d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f14b32c273e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f14c4fa21c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f14c4fa1783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f14c502d2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f14c502d0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f14c50775e9 std::function<>::operator()() at ??:0
@ 0x7f14c4fa5cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f14c4fa3c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f14c502b610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f14c502b439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f14b21bb345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f14b21be91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f14b21be799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f14b2199eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f14b54736db start_thread at ??:0
@ 0x7f14ae40961f clone at ??:0
W20260927 15:17:33.287945 26997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:33.288275 26997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:33.288404 26997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:33.295940 26997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:33.296064 26997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:33.296317 26997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:17:33.306850 26997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:33097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=45933
--tserver_master_addrs=127.10.33.126:39747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:17:33.308192 26997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:33.309751 26997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:33.323423 27044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:33.323408 27046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:33.323613 27043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:33.324777 26997 server_base.cc:1034] running on GCE node
I20260927 15:17:33.325431 26997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:33.326574 26997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:33.327813 26997 hybrid_clock.cc:648] HybridClock initialized: now 1790522253327759 us; error 50 us; skew 500 ppm
I20260927 15:17:33.328122 26997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:17:33 dist-test-slave-f5h6 krb5kdc[25844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522253, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:33.331389 26997 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:17:33.332963 26997 webserver.cc:460] Webserver started at http://127.10.33.65:45933/ using document root <none> and password file <none>
I20260927 15:17:33.333642 26997 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:33.333771 26997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:33.339201 26997 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:17:33.341811 27053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:33.343374 26997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:17:33.343528 26997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "64e324de2d014f7980bf068d1cf49792"
format_stamp: "Formatted at 2026-09-27 15:16:47 on dist-test-slave-f5h6"
server_key: "c364214df20d066e5ebf77b0b1d766e8"
server_key_iv: "c464ba63cefedcd00bc822a8213c42f9"
server_key_version: "encryptionkey@0"
I20260927 15:17:33.344492 26997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:33.364050 26997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:33.368286 26997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:33.368546 26997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:33.369540 26997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:33.371346 27060 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260927 15:17:33.376991 27060 ts_tablet_manager.cc:1852] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 15:17:33.378046 27060 ts_tablet_manager.cc:1905] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:33.386062 27060 ts_tablet_manager.cc:1918] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:33.386400 27060 ts_tablet_manager.cc:1939] T 592934b040f448f88fa4ff473b4690b7 P 64e324de2d014f7980bf068d1cf49792: Deleting consensus metadata
I20260927 15:17:33.386902 26997 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 15:17:33.387058 26997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260927 15:17:33.387185 26997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:33.387262 26997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:33.419109 26997 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:33097
I20260927 15:17:33.419221 27166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:33097 every 8 connection(s)
I20260927 15:17:33.420383 26997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:17:33.425251 27169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:33.423402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.65:43399)
0927 15:17:33.424139 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:33.424377 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":289,"threads_started":1}
W20260927 15:17:33.425920 27167 heartbeater.cc:644] Failed to heartbeat to 127.10.33.126:39747 (0 consecutive failures): Network error: Failed to ping master at 127.10.33.126:39747: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
I20260927 15:17:33.426740 27169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:33.426403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.65:58673)
0927 15:17:33.426554 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:33.426632 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 15:17:33.427619 27169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:33.427286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.65:39111)
0927 15:17:33.427426 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:33.427499 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 15:17:33.427868 27167 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 15:17:33.428108 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 26997
I20260927 15:17:33.428714 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:35461
--local_ip_for_outbound_sockets=127.10.33.66
--tserver_master_addrs=127.10.33.126:39747
--webserver_port=36465
--webserver_interface=127.10.33.66
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:33.751469 27170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:33.751761 27170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:33.751880 27170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:33.759186 27170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:33.759312 27170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:33.759466 27170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:17:33.769174 27170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:35461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=36465
--tserver_master_addrs=127.10.33.126:39747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:17:33.770503 27170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:33.771780 27170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:33.784354 27175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:33.784404 27178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:33.784463 27176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:33.785391 27170 server_base.cc:1034] running on GCE node
I20260927 15:17:33.785990 27170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:33.786893 27170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:33.788084 27170 hybrid_clock.cc:648] HybridClock initialized: now 1790522253788040 us; error 36 us; skew 500 ppm
I20260927 15:17:33.788401 27170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:17:33 dist-test-slave-f5h6 krb5kdc[25844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522253, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:33.791407 27170 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:17:33.792773 27170 webserver.cc:460] Webserver started at http://127.10.33.66:36465/ using document root <none> and password file <none>
I20260927 15:17:33.793422 27170 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:33.793535 27170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:33.798766 27170 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:17:33.801258 27185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:33.802536 27170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:17:33.802685 27170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "ccece2bca11944899ae41baa61d49801"
format_stamp: "Formatted at 2026-09-27 15:16:48 on dist-test-slave-f5h6"
server_key: "d0920b366530a4e93fbfeb89a670ceee"
server_key_iv: "2e7bc8d665cd29f159ef31ca959ca4fb"
server_key_version: "encryptionkey@0"
I20260927 15:17:33.803637 27170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:33.828925 27170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:33.832942 27170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:33.833220 27170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:33.834246 27170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:33.835598 27170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:33.835700 27170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:33.835808 27170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:33.835877 27170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:33.866856 27170 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:35461
I20260927 15:17:33.866936 27297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:35461 every 8 connection(s)
I20260927 15:17:33.868311 27170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:17:33.871497 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 27170
I20260927 15:17:33.872154 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:32889
--local_ip_for_outbound_sockets=127.10.33.67
--tserver_master_addrs=127.10.33.126:39747
--webserver_port=33701
--webserver_interface=127.10.33.67
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:17:33.873313 27300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:33.871239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.66:55011)
0927 15:17:33.872118 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:33.872436 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":399,"threads_started":1}
W20260927 15:17:33.874058 27298 heartbeater.cc:644] Failed to heartbeat to 127.10.33.126:39747 (0 consecutive failures): Network error: Failed to ping master at 127.10.33.126:39747: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
I20260927 15:17:33.874938 27300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:33.874608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.66:33299)
0927 15:17:33.874759 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:33.874829 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 15:17:33.875803 27300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:33.875472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.66:44873)
0927 15:17:33.875627 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:33.875698 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 15:17:33.876013 27298 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:17:34.196036 27301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:34.196326 27301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:34.196442 27301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:34.203759 27301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:34.203868 27301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:34.204005 27301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:17:34.213845 27301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:32889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=33701
--tserver_master_addrs=127.10.33.126:39747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:17:34.215117 27301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:34.216375 27301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:34.229174 27309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:34.229235 27306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:34.229211 27307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:34.229898 27301 server_base.cc:1034] running on GCE node
I20260927 15:17:34.230492 27301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:34.231420 27301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:34.232605 27301 hybrid_clock.cc:648] HybridClock initialized: now 1790522254232540 us; error 59 us; skew 500 ppm
I20260927 15:17:34.232914 27301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:17:34 dist-test-slave-f5h6 krb5kdc[25844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522254, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:34.235858 27301 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:17:34.237279 27301 webserver.cc:460] Webserver started at http://127.10.33.67:33701/ using document root <none> and password file <none>
I20260927 15:17:34.237973 27301 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:34.238078 27301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:34.243614 27301 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:17:34.246826 27316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:34.248569 27301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:17:34.248749 27301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "058c7d5d33ad46af8d87df79504209c5"
format_stamp: "Formatted at 2026-09-27 15:16:48 on dist-test-slave-f5h6"
server_key: "e29ea2a1e4c62e9961ad206000a5d107"
server_key_iv: "5cc9d537c274292370bc0518228972ac"
server_key_version: "encryptionkey@0"
I20260927 15:17:34.249811 27301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:34.262132 27301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:34.266898 27301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:34.267166 27301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:34.268258 27301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:34.269654 27301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:34.269752 27301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:34.269881 27301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:34.269956 27301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 15:17:34.301676 27301 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:32889
I20260927 15:17:34.301760 27428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:32889 every 8 connection(s)
I20260927 15:17:34.303128 27301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:17:34.305335 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 27301
I20260927 15:17:34.308207 27432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:34.306097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.67:33055)
0927 15:17:34.307022 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:34.307341 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":341,"threads_started":1}
W20260927 15:17:34.308838 27429 heartbeater.cc:644] Failed to heartbeat to 127.10.33.126:39747 (0 consecutive failures): Network error: Failed to ping master at 127.10.33.126:39747: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
I20260927 15:17:34.309633 27432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:34.309321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.67:33969)
0927 15:17:34.309463 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:34.309530 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 15:17:34.310473 27432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:34.310152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.67:51225)
0927 15:17:34.310304 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:34.310373 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 15:17:34.310691 27429 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 15:17:34.429433 27433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:34.428587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.65:39605)
0927 15:17:34.429099 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:34.429202 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":236,"threads_started":1}
I20260927 15:17:34.877627 27434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:34.876682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.66:42875)
0927 15:17:34.877207 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:34.877315 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":270,"threads_started":1}
I20260927 15:17:35.312096 27435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:35.311269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.67:40661)
0927 15:17:35.311783 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:35.311870 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":217,"threads_started":1}
I20260927 15:17:35.431221 27436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:35.430463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.65:42075)
0927 15:17:35.430956 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:35.431048 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":280,"threads_started":1}
I20260927 15:17:35.879299 27437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:35.878580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.66:58111)
0927 15:17:35.879049 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:35.879129 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":244,"threads_started":1}
I20260927 15:17:36.313814 27438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:36.313051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.67:35191)
0927 15:17:36.313517 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:36.313607 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":230,"threads_started":1}
I20260927 15:17:36.432868 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:36.432119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.65:34465)
0927 15:17:36.432613 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:36.432705 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":249,"threads_started":1}
I20260927 15:17:36.880949 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:36.880131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.66:34781)
0927 15:17:36.880661 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:36.880755 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":276,"threads_started":1}
I20260927 15:17:37.315586 27441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:37.314858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.67:45259)
0927 15:17:37.315318 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:37.315406 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":240,"threads_started":1}
I20260927 15:17:37.434448 27442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:37.433711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.65:33859)
0927 15:17:37.434182 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:37.434269 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":234,"threads_started":1}
I20260927 15:17:37.882706 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:37.882007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.66:56561)
0927 15:17:37.882431 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:37.882512 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":199,"threads_started":1}
I20260927 15:17:38.317412 27444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:38.316655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.67:37515)
0927 15:17:38.317124 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:38.317214 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":226,"threads_started":1}
I20260927 15:17:38.436054 27445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:38.435361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.65:38769)
0927 15:17:38.435823 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:38.435905 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":230,"threads_started":1}
I20260927 15:17:38.884479 27446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:38.883703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.66:37637)
0927 15:17:38.884226 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:38.884309 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":266,"threads_started":1}
I20260927 15:17:39.319312 27447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:39.318527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.67:44813)
0927 15:17:39.319050 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:39.319141 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":288,"threads_started":1}
I20260927 15:17:39.437702 27448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:39.437002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.65:43637)
0927 15:17:39.437441 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:39.437520 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":227,"threads_started":1}
I20260927 15:17:39.886498 27449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:39.885558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.66:50045)
0927 15:17:39.886192 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:39.886324 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":337,"threads_started":1}
I20260927 15:17:40.321063 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:40.320251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.67:33901)
0927 15:17:40.320778 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:40.320888 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":238,"threads_started":1}
I20260927 15:17:40.439630 27451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:40.438700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.65:35397)
0927 15:17:40.439331 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:40.439417 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":316,"threads_started":1}
I20260927 15:17:40.888362 27452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:40.887547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.66:34573)
0927 15:17:40.888042 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:40.888130 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":214,"threads_started":1}
I20260927 15:17:41.323067 27453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:41.322142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.67:37821)
0927 15:17:41.322737 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:41.322841 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":247,"threads_started":1}
I20260927 15:17:41.441344 27454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:41.440544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.65:36519)
0927 15:17:41.441051 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:41.441147 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":251,"threads_started":1}
I20260927 15:17:41.890112 27455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:41.889340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.66:44809)
0927 15:17:41.889823 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:41.889947 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":252,"threads_started":1}
I20260927 15:17:42.324952 27456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:42.324220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.67:34111)
0927 15:17:42.324695 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:42.324783 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":242,"threads_started":1}
I20260927 15:17:42.443130 27457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:42.442362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.65:38341)
0927 15:17:42.442889 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:42.442981 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":256,"threads_started":1}
I20260927 15:17:42.892197 27458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:42.891243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.66:33023)
0927 15:17:42.891906 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:42.892013 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":279,"threads_started":1}
I20260927 15:17:43.326748 27459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:43.326014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.67:52505)
0927 15:17:43.326492 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:43.326577 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":232,"threads_started":1}
I20260927 15:17:43.444926 27460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:43.444095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.65:49749)
0927 15:17:43.444633 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:43.444715 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":296,"threads_started":1}
I20260927 15:17:43.894274 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:43.893403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.66:59091)
0927 15:17:43.893986 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:43.894081 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":300,"threads_started":1}
I20260927 15:17:44.328441 27462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:44.327679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.67:48229)
0927 15:17:44.328160 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:44.328250 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":257,"threads_started":1}
I20260927 15:17:44.446511 27463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:44.445791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.65:47327)
0927 15:17:44.446253 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:44.446339 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":262,"threads_started":1}
I20260927 15:17:44.896224 27464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:44.895340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.66:47047)
0927 15:17:44.895880 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:44.895987 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":227,"threads_started":1}
I20260927 15:17:45.330385 27465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:45.329482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.67:57493)
0927 15:17:45.330113 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:45.330200 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":380,"threads_started":1}
I20260927 15:17:45.448410 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:45.447572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.65:53073)
0927 15:17:45.448099 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:45.448197 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":279,"threads_started":1}
I20260927 15:17:45.898370 27467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:45.897462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.66:51411)
0927 15:17:45.898057 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:45.898197 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":276,"threads_started":1}
I20260927 15:17:46.332055 27468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:46.331313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.67:33215)
0927 15:17:46.331755 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:46.331902 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":228,"threads_started":1}
I20260927 15:17:46.450348 27469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:46.449458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.65:57329)
0927 15:17:46.450045 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:46.450167 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":319,"threads_started":1}
I20260927 15:17:46.900300 27470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:46.899413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.66:43245)
0927 15:17:46.900003 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:46.900109 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":246,"threads_started":1}
I20260927 15:17:47.334218 27471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:47.333241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.67:45087)
0927 15:17:47.333765 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:47.334037 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":253,"threads_started":1}
I20260927 15:17:47.452061 27472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:47.451188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.65:34223)
0927 15:17:47.451711 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:47.451857 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":270,"threads_started":1}
I20260927 15:17:47.902161 27473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:47.901335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.66:44579)
0927 15:17:47.901893 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:47.901978 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":305,"threads_started":1}
I20260927 15:17:48.336045 27474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:48.335216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.67:34011)
0927 15:17:48.335759 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:48.335859 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":237,"threads_started":1}
I20260927 15:17:48.454133 27475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:48.453222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.65:60905)
0927 15:17:48.453690 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:48.453897 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":216,"threads_started":1}
I20260927 15:17:48.904119 27476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:48.903329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.66:47717)
0927 15:17:48.903780 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:48.903912 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":230,"threads_started":1}
I20260927 15:17:49.309955 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 26997
I20260927 15:17:49.334151 27477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:49.311041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39628 (local address 127.10.33.65:33097)
0927 15:17:49.311712 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:49.311733 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:49.311783 (+ 50us) server_negotiation.cc:407] Connection header received
0927 15:17:49.313604 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:49.313616 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:49.314128 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:49.314448 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:49.316132 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:49.317603 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:49.319372 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:49.320270 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:49.320537 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:49.327054 (+ 6517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:49.327065 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:49.327084 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:49.327145 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:49.330452 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:49.331325 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:49.331333 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:49.331350 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:49.331556 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:49.331963 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:49.331967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:49.331970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:49.332633 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:49.332879 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:49.333391 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 15:17:49.333770 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":299,"threads_started":1}
I20260927 15:17:49.335721 27162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:17:49.338238 27479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:49.337258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39747 (local address 127.10.33.67:40173)
0927 15:17:49.337881 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:49.338012 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.126:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":340,"threads_started":1}
I20260927 15:17:49.421999 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 26997
I20260927 15:17:49.436723 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 27170
I20260927 15:17:49.454927 27480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:49.437625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57878 (local address 127.10.33.66:35461)
0927 15:17:49.438192 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:49.438206 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:49.438241 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:17:49.439666 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:49.439676 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:49.440095 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:49.440377 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:49.441701 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:49.443136 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:49.444479 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:49.445252 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:49.445401 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:49.448670 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:49.448679 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:49.448701 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:49.448749 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:49.451853 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:49.452513 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:49.452520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:49.452526 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:49.452694 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:49.453010 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:49.453013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:49.453015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:49.453545 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:49.453817 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:49.454168 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 15:17:49.454561 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":290,"threads_started":1}
I20260927 15:17:49.456384 27293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:17:49.540390 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 27170
I20260927 15:17:49.554486 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 27301
I20260927 15:17:49.571821 27482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:49.555507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37814 (local address 127.10.33.67:32889)
0927 15:17:49.556076 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:49.556091 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:49.556130 (+ 39us) server_negotiation.cc:407] Connection header received
0927 15:17:49.557733 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:49.557742 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:49.558340 (+ 598us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:49.558628 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:49.559966 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:49.561217 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:49.562539 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:49.563209 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:49.563351 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:49.566025 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:49.566035 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:49.566051 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:49.566095 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:49.568863 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:49.569526 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:49.569531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:49.569538 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:49.569693 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:49.570012 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:49.570015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:49.570017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:49.570534 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:49.570751 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:49.571141 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 15:17:49.571492 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":262,"threads_started":1}
I20260927 15:17:49.573192 27424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:17:49.655107 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 27301
2026-09-27T15:17:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64270 ms)
[----------] 30 tests from SecurityITest (301222 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:17:49 dist-test-slave-f5h6 krb5kdc[27486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:17:49 dist-test-slave-f5h6 krb5kdc[27486](info): set up 2 sockets
Sep 27 15:17:49 dist-test-slave-f5h6 krb5kdc[27486](info): commencing operation
krb5kdc: starting...
W20260927 15:17:50.677776 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:17:50 dist-test-slave-f5h6 krb5kdc[27486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:17:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:17:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:17:50.921227 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:45085
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:46131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:45085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:51.272393 27499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:51.272725 27499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:51.272827 27499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:51.280437 27499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:17:51.280560 27499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:51.280606 27499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:51.280661 27499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:17:51.280704 27499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:17:51.290917 27499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:46131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:45085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:45085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:17:51.292261 27499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:51.293514 27499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:51.305774 27504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:51.305698 27507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:51.305738 27505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:51.306782 27499 server_base.cc:1034] running on GCE node
I20260927 15:17:51.307613 27499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:51.309176 27499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:51.310413 27499 hybrid_clock.cc:648] HybridClock initialized: now 1790522271310355 us; error 44 us; skew 500 ppm
I20260927 15:17:51.310777 27499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:17:51 dist-test-slave-f5h6 krb5kdc[27486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522271, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:51.314265 27499 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:17:51.316057 27499 webserver.cc:460] Webserver started at http://127.10.33.126:34541/ using document root <none> and password file <none>
I20260927 15:17:51.316820 27499 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:51.316944 27499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:51.317242 27499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:51.320477 27499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "6e298c1e98c94d5a97ea772914b3a473"
format_stamp: "Formatted at 2026-09-27 15:17:51 on dist-test-slave-f5h6"
server_key: "9a065bee8fdf8158d0a34520d5579e43"
server_key_iv: "7131b788f9dd06faa37305d3df929db3"
server_key_version: "encryptionkey@0"
I20260927 15:17:51.321274 27499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "6e298c1e98c94d5a97ea772914b3a473"
format_stamp: "Formatted at 2026-09-27 15:17:51 on dist-test-slave-f5h6"
server_key: "9a065bee8fdf8158d0a34520d5579e43"
server_key_iv: "7131b788f9dd06faa37305d3df929db3"
server_key_version: "encryptionkey@0"
I20260927 15:17:51.327109 27499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 15:17:51.331025 27514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:51.332532 27499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:17:51.332736 27499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "6e298c1e98c94d5a97ea772914b3a473"
format_stamp: "Formatted at 2026-09-27 15:17:51 on dist-test-slave-f5h6"
server_key: "9a065bee8fdf8158d0a34520d5579e43"
server_key_iv: "7131b788f9dd06faa37305d3df929db3"
server_key_version: "encryptionkey@0"
I20260927 15:17:51.333000 27499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:51.350750 27499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:51.355611 27499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:51.355895 27499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:51.374640 27499 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:45085
I20260927 15:17:51.374644 27565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:45085 every 8 connection(s)
I20260927 15:17:51.376083 27499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:17:51.380179 27566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:51.386086 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 27499
I20260927 15:17:51.386317 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:17:51.387097 10373 external_mini_cluster.cc:1428] Setting key b02c71c4a5f5ab72fa896f0aff7db469
I20260927 15:17:51.392515 27566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473: Bootstrap starting.
I20260927 15:17:51.395714 27566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:51.396736 27566 log.cc:824] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:51.400020 27566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473: No bootstrap required, opened a new log
Sep 27 15:17:51 dist-test-slave-f5h6 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:17:51.409824 27566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e298c1e98c94d5a97ea772914b3a473" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45085 } }
I20260927 15:17:51.410386 27566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:51.410511 27566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e298c1e98c94d5a97ea772914b3a473, State: Initialized, Role: FOLLOWER
I20260927 15:17:51.411247 27566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e298c1e98c94d5a97ea772914b3a473" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45085 } }
I20260927 15:17:51.411557 27566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:17:51.411716 27566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:17:51.411902 27566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:51.412592 27569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:51.389443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41170 (local address 127.10.33.126:45085)
0927 15:17:51.390392 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:51.390405 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:51.390459 (+ 54us) server_negotiation.cc:407] Connection header received
0927 15:17:51.392197 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:51.392221 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:51.392655 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:51.393035 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:51.394612 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:51.395935 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:51.397532 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:51.398566 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:51.398763 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:51.403058 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:51.403069 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:51.403095 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:51.403146 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:51.407228 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:51.408280 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:51.408291 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:51.408302 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:51.408575 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:51.409069 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:51.409074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:51.409077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:51.409843 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:51.410160 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:51.410625 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 15:17:51.411116 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":477,"threads_started":1}
I20260927 15:17:51.414974 27566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e298c1e98c94d5a97ea772914b3a473" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45085 } }
I20260927 15:17:51.415517 27566 leader_election.cc:302] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e298c1e98c94d5a97ea772914b3a473; no voters:
I20260927 15:17:51.416200 27566 leader_election.cc:288] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:17:51.416296 27571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:17:51.416935 27571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473 [term 1 LEADER]: Becoming Leader. State: Replica: 6e298c1e98c94d5a97ea772914b3a473, State: Running, Role: LEADER
I20260927 15:17:51.417552 27571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e298c1e98c94d5a97ea772914b3a473" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45085 } }
I20260927 15:17:51.418551 27566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:17:51.420769 27572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e298c1e98c94d5a97ea772914b3a473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e298c1e98c94d5a97ea772914b3a473" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45085 } } }
I20260927 15:17:51.420737 27573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e298c1e98c94d5a97ea772914b3a473. Latest consensus state: current_term: 1 leader_uuid: "6e298c1e98c94d5a97ea772914b3a473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e298c1e98c94d5a97ea772914b3a473" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45085 } } }
I20260927 15:17:51.421190 27572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473 [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:51.421191 27573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473 [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:51.424907 27577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:17:51.433439 27577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:17:51.443348 27577 catalog_manager.cc:1347] Generated new cluster ID: 4cf4691a5dd44796bc2464da360ca0a6
I20260927 15:17:51.443498 27577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:17:51.461596 27577 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:17:51.463527 27577 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:17:51.474314 27577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473: Generated new TSK 0
I20260927 15:17:51.475448 27577 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.33.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
I20260927 15:17:51.535131 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:45085
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:46131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:51.872495 27592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:51.872789 27592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:51.872910 27592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:51.880858 27592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:51.880995 27592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:51.881150 27592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:17:51.891623 27592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:46131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:45085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:17:51.892962 27592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:51.894363 27592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:51.908095 27598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:51.908061 27600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:51.907996 27597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:51.908751 27592 server_base.cc:1034] running on GCE node
I20260927 15:17:51.909418 27592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:51.910522 27592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:51.911782 27592 hybrid_clock.cc:648] HybridClock initialized: now 1790522271911690 us; error 79 us; skew 500 ppm
I20260927 15:17:51.912142 27592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:17:51 dist-test-slave-f5h6 krb5kdc[27486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522271, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:51.915675 27592 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:17:51.917411 27592 webserver.cc:460] Webserver started at http://127.10.33.65:34637/ using document root <none> and password file <none>
I20260927 15:17:51.918203 27592 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:51.918332 27592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:51.918671 27592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:51.922029 27592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "69612d37448a46fcb422801ee1d1da1d"
format_stamp: "Formatted at 2026-09-27 15:17:51 on dist-test-slave-f5h6"
server_key: "07940680b688a68765543d12514a8759"
server_key_iv: "26982909ed04a63208233556f45d5e3d"
server_key_version: "encryptionkey@0"
I20260927 15:17:51.922958 27592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "69612d37448a46fcb422801ee1d1da1d"
format_stamp: "Formatted at 2026-09-27 15:17:51 on dist-test-slave-f5h6"
server_key: "07940680b688a68765543d12514a8759"
server_key_iv: "26982909ed04a63208233556f45d5e3d"
server_key_version: "encryptionkey@0"
I20260927 15:17:51.928869 27592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:17:51.932720 27607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:51.934293 27592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:17:51.934466 27592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "69612d37448a46fcb422801ee1d1da1d"
format_stamp: "Formatted at 2026-09-27 15:17:51 on dist-test-slave-f5h6"
server_key: "07940680b688a68765543d12514a8759"
server_key_iv: "26982909ed04a63208233556f45d5e3d"
server_key_version: "encryptionkey@0"
I20260927 15:17:51.934667 27592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:51.955205 27592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:51.959813 27592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:51.960122 27592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:51.961277 27592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:51.962821 27592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:51.962900 27592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:51.963006 27592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:51.963064 27592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:52.000062 27592 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:45123
I20260927 15:17:52.000120 27719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:45123 every 8 connection(s)
I20260927 15:17:52.001379 27592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:17:52.011883 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 27592
I20260927 15:17:52.012109 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:17:52.012840 10373 external_mini_cluster.cc:1428] Setting key 2dbe2caa9ca28cad4f7e17387b60ad73
Sep 27 15:17:52 dist-test-slave-f5h6 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522271, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:17:52.022485 27722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:52.004015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:49277 (local address 127.10.33.126:45085)
0927 15:17:52.004515 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:52.004521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:52.005652 (+ 1131us) server_negotiation.cc:407] Connection header received
0927 15:17:52.007543 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:52.007548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:52.007634 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:52.007744 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:52.009492 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:52.010355 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:52.012376 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:52.012826 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:52.013027 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:52.016735 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:52.016740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:52.016744 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:52.016784 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:52.019366 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:52.020145 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:52.020151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:52.020155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:52.020275 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:52.020691 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:52.020696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:52.020700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:52.020971 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:52.021101 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:52.021819 (+ 718us) server_negotiation.cc:299] Negotiation successful
0927 15:17:52.022084 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":236,"threads_started":1}
I20260927 15:17:52.023461 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:52.004519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:45085 (local address 127.10.33.65:49277)
0927 15:17:52.005465 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:52.005509 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:52.007236 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:52.008050 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:52.008060 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:52.008665 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:52.009288 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:52.009309 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:52.010507 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:52.010512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:52.012162 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:52.012180 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:52.013096 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:52.013104 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:52.013418 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:52.014144 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:52.014175 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:52.016499 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:52.019504 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:52.019514 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:52.019532 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:52.020033 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:52.020377 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:52.020382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:52.020385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:52.020586 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:52.021093 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:52.021101 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:52.021454 (+ 353us) client_negotiation.cc:769] Sending connection context
0927 15:17:52.021714 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 15:17:52.022307 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":350,"threads_started":1}
I20260927 15:17:52.025094 27720 heartbeater.cc:347] Connected to a master server at 127.10.33.126:45085
I20260927 15:17:52.025404 27720 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:52.026183 27720 heartbeater.cc:511] Master 127.10.33.126:45085 requested a full tablet report, sending...
I20260927 15:17:52.028199 27531 ts_manager.cc:194] Registered new tserver with Master: 69612d37448a46fcb422801ee1d1da1d (127.10.33.65:45123)
I20260927 15:17:52.029693 27531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:49277
WARNING: no policy specified for kudu/127.10.33.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
I20260927 15:17:52.070484 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:45085
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:46131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:52.399237 27726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:52.399580 27726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:52.399724 27726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:52.407573 27726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:52.407729 27726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:52.407915 27726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:17:52.419083 27726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:46131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:45085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:17:52.420496 27726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:52.421849 27726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:52.435048 27732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:52.435081 27734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:52.435016 27731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:52.436197 27726 server_base.cc:1034] running on GCE node
I20260927 15:17:52.436851 27726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:52.438094 27726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:52.439337 27726 hybrid_clock.cc:648] HybridClock initialized: now 1790522272439283 us; error 47 us; skew 500 ppm
I20260927 15:17:52.439699 27726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:17:52 dist-test-slave-f5h6 krb5kdc[27486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522272, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:52.442994 27726 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:17:52.445011 27726 webserver.cc:460] Webserver started at http://127.10.33.66:44173/ using document root <none> and password file <none>
I20260927 15:17:52.445729 27726 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:52.445876 27726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:52.446190 27726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:52.449295 27726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "069c358ffb33448a973721e31b388822"
format_stamp: "Formatted at 2026-09-27 15:17:52 on dist-test-slave-f5h6"
server_key: "d9c80be0b54ea412e3217e9a658d3c61"
server_key_iv: "3922899bcc944ccc1caa53d06d5ba2f6"
server_key_version: "encryptionkey@0"
I20260927 15:17:52.450152 27726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "069c358ffb33448a973721e31b388822"
format_stamp: "Formatted at 2026-09-27 15:17:52 on dist-test-slave-f5h6"
server_key: "d9c80be0b54ea412e3217e9a658d3c61"
server_key_iv: "3922899bcc944ccc1caa53d06d5ba2f6"
server_key_version: "encryptionkey@0"
I20260927 15:17:52.455591 27726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 15:17:52.459421 27741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:52.460742 27726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:17:52.460881 27726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "069c358ffb33448a973721e31b388822"
format_stamp: "Formatted at 2026-09-27 15:17:52 on dist-test-slave-f5h6"
server_key: "d9c80be0b54ea412e3217e9a658d3c61"
server_key_iv: "3922899bcc944ccc1caa53d06d5ba2f6"
server_key_version: "encryptionkey@0"
I20260927 15:17:52.461063 27726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:52.496443 27726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:52.501060 27726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:52.501331 27726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:52.502417 27726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:52.503859 27726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:52.503937 27726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:52.504046 27726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:52.504096 27726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:52.534986 27726 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:37397
I20260927 15:17:52.535079 27853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:37397 every 8 connection(s)
I20260927 15:17:52.536227 27726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:17:52.545727 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 27726
I20260927 15:17:52.545953 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:17:52.546458 10373 external_mini_cluster.cc:1428] Setting key f3e221ca9f648e38c90b54b04fa7164b
Sep 27 15:17:52 dist-test-slave-f5h6 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522272, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:17:52.556790 27856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:52.538673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:36011 (local address 127.10.33.126:45085)
0927 15:17:52.539105 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:52.539110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:52.540086 (+ 976us) server_negotiation.cc:407] Connection header received
0927 15:17:52.541876 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:52.541883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:52.541968 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:52.542062 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:52.543562 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:52.544285 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:52.546151 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:52.546636 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:52.546739 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:52.550915 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:52.550920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:52.550923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:52.550956 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:52.553405 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:52.554312 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:52.554317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:52.554320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:52.554471 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:52.554949 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:52.554952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:52.554954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:52.555252 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:52.555407 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:52.556240 (+ 833us) server_negotiation.cc:299] Negotiation successful
0927 15:17:52.556431 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":217,"threads_started":1}
I20260927 15:17:52.557739 27857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:52.539243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:45085 (local address 127.10.33.66:36011)
0927 15:17:52.539930 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:52.539974 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:52.541596 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:52.542312 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:52.542320 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:52.542820 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:52.543391 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:52.543409 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:52.544451 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:52.544455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:52.545980 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:52.545996 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:52.546895 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:52.546903 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:52.547150 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:52.547821 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:52.547859 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:52.550661 (+ 2802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:52.553551 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:52.553560 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:52.553579 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:52.554183 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:52.554583 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:52.554587 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:52.554590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:52.554824 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:52.555401 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:52.555409 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:52.555830 (+ 421us) client_negotiation.cc:769] Sending connection context
0927 15:17:52.556132 (+ 302us) client_negotiation.cc:240] Negotiation successful
0927 15:17:52.556657 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":253,"threads_started":1}
I20260927 15:17:52.559176 27854 heartbeater.cc:347] Connected to a master server at 127.10.33.126:45085
I20260927 15:17:52.559469 27854 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:52.560148 27854 heartbeater.cc:511] Master 127.10.33.126:45085 requested a full tablet report, sending...
I20260927 15:17:52.561582 27531 ts_manager.cc:194] Registered new tserver with Master: 069c358ffb33448a973721e31b388822 (127.10.33.66:37397)
I20260927 15:17:52.562448 27531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:36011
WARNING: no policy specified for kudu/127.10.33.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
I20260927 15:17:52.601076 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:45085
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:46131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:52.933676 27860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:52.934042 27860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:52.934185 27860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:52.941732 27860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:52.941905 27860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:52.942063 27860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:17:52.951958 27860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:46131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:45085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:17:52.953290 27860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:52.954862 27860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:52.968266 27868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:52.968329 27865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:52.968909 27866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:52.969264 27860 server_base.cc:1034] running on GCE node
I20260927 15:17:52.969903 27860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:52.970969 27860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:52.972172 27860 hybrid_clock.cc:648] HybridClock initialized: now 1790522272972110 us; error 51 us; skew 500 ppm
I20260927 15:17:52.972486 27860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:17:52 dist-test-slave-f5h6 krb5kdc[27486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522272, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:52.975759 27860 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:17:52.977321 27860 webserver.cc:460] Webserver started at http://127.10.33.67:33469/ using document root <none> and password file <none>
I20260927 15:17:52.978106 27860 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:52.978235 27860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:52.978542 27860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:52.981679 27860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "f76a464efff645e7b6bab60dfd4baee6"
format_stamp: "Formatted at 2026-09-27 15:17:52 on dist-test-slave-f5h6"
server_key: "d70e0b18eb26a59df993e4353d14743e"
server_key_iv: "4df5168786141475a6b35b4307a51343"
server_key_version: "encryptionkey@0"
I20260927 15:17:52.982522 27860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "f76a464efff645e7b6bab60dfd4baee6"
format_stamp: "Formatted at 2026-09-27 15:17:52 on dist-test-slave-f5h6"
server_key: "d70e0b18eb26a59df993e4353d14743e"
server_key_iv: "4df5168786141475a6b35b4307a51343"
server_key_version: "encryptionkey@0"
I20260927 15:17:52.987872 27860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:17:52.991645 27875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:52.993196 27860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:17:52.993379 27860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "f76a464efff645e7b6bab60dfd4baee6"
format_stamp: "Formatted at 2026-09-27 15:17:52 on dist-test-slave-f5h6"
server_key: "d70e0b18eb26a59df993e4353d14743e"
server_key_iv: "4df5168786141475a6b35b4307a51343"
server_key_version: "encryptionkey@0"
I20260927 15:17:52.993626 27860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:53.016937 27860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:53.021591 27860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:53.021903 27860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:53.023100 27860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:53.024674 27860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:53.024757 27860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:53.024845 27860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:53.024889 27860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:53.033099 27720 heartbeater.cc:503] Master 127.10.33.126:45085 was elected leader, sending a full tablet report...
I20260927 15:17:53.055635 27860 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:37771
I20260927 15:17:53.055697 27987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:37771 every 8 connection(s)
I20260927 15:17:53.056938 27860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:17:53.065318 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 27860
I20260927 15:17:53.065506 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:17:53.066073 10373 external_mini_cluster.cc:1428] Setting key fd242132c10c8fb7d3b9ce1f173e5e14
Sep 27 15:17:53 dist-test-slave-f5h6 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522272, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:17:53.077746 27990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:53.059491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:48799 (local address 127.10.33.126:45085)
0927 15:17:53.059913 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:53.059921 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:53.061126 (+ 1205us) server_negotiation.cc:407] Connection header received
0927 15:17:53.063010 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:53.063015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:53.063107 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:53.063207 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:53.064884 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.066082 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.067947 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.068434 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.068565 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:53.072107 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:53.072112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:53.072115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:53.072155 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:53.074667 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:53.075450 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:53.075454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:53.075457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:53.075585 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:53.076015 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:53.076019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:53.076021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:53.076287 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:53.076437 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:53.077122 (+ 685us) server_negotiation.cc:299] Negotiation successful
0927 15:17:53.077360 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":185,"threads_started":1}
I20260927 15:17:53.078711 27991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:53.060070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:45085 (local address 127.10.33.67:48799)
0927 15:17:53.060932 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:53.060980 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:53.062738 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:53.063438 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:53.063447 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:53.064015 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:53.064661 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:53.064682 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.066254 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.066258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:53.067792 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:53.067810 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.068674 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.068682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:53.068949 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:53.069602 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:53.069629 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:53.071870 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:53.074827 (+ 2957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:53.074842 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:53.074859 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:53.075320 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:53.075692 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:53.075697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:53.075700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:53.075908 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:53.076412 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:53.076420 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:53.076767 (+ 347us) client_negotiation.cc:769] Sending connection context
0927 15:17:53.077025 (+ 258us) client_negotiation.cc:240] Negotiation successful
0927 15:17:53.077566 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":405,"threads_started":1}
I20260927 15:17:53.080212 27988 heartbeater.cc:347] Connected to a master server at 127.10.33.126:45085
I20260927 15:17:53.080547 27988 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:53.081447 27988 heartbeater.cc:511] Master 127.10.33.126:45085 requested a full tablet report, sending...
I20260927 15:17:53.082935 27531 ts_manager.cc:194] Registered new tserver with Master: f76a464efff645e7b6bab60dfd4baee6 (127.10.33.67:37771)
I20260927 15:17:53.083758 27531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:48799
I20260927 15:17:53.094777 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:17:53.113807 27990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:53.099956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41178 (local address 127.10.33.126:45085)
0927 15:17:53.100185 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:53.100193 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:53.100615 (+ 422us) server_negotiation.cc:407] Connection header received
0927 15:17:53.100776 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:53.100780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:53.100856 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:53.100951 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:53.102530 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.103267 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.104770 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.105242 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.105383 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:53.108033 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:53.108039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:53.108043 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:53.108082 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:53.110876 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:53.111723 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:53.111727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:53.111729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:53.111951 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:53.112371 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:53.112376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:53.112378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:53.112758 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:53.112968 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:53.113310 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 15:17:53.113543 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 15:17:53.118319 27531 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:17:53.125241 27531 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:17:53.148933 28001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:53.137986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46894 (local address 127.10.33.66:37397)
0927 15:17:53.138553 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:53.138562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:53.139209 (+ 647us) server_negotiation.cc:407] Connection header received
0927 15:17:53.139635 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:53.139641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:53.139884 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:53.140062 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:53.140971 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.142536 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.146075 (+ 3539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.147590 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.147727 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:53.147940 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:53.148401 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 15:17:53.148595 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":519,"threads_started":1}
I20260927 15:17:53.149006 28002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:53.137955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:37397 (local address 127.0.0.1:46894)
0927 15:17:53.139049 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:53.139098 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:53.139379 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:53.140382 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:53.140389 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:53.140417 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:53.140776 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:53.140790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.142712 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.142717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:53.145921 (+ 3204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:53.145936 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.147849 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.147858 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:53.148130 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:53.148187 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 15:17:53.148334 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 15:17:53.148557 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":61,"thread_start_us":732,"threads_started":1}
I20260927 15:17:53.150557 28005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:53.138386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:45123 (local address 127.0.0.1:38064)
0927 15:17:53.140061 (+ 1675us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:53.140077 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:53.140256 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:53.141091 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:53.141096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:53.141118 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:53.141506 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:53.141518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.143357 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.143362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:53.145924 (+ 2562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:53.145936 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.149710 (+ 3774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.149716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:53.149931 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:53.149964 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:17:53.150059 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 15:17:53.150191 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1558,"mutex_wait_us":76,"thread_start_us":5629,"threads_started":1}
I20260927 15:17:53.150604 28004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:53.138919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38064 (local address 127.10.33.65:45123)
0927 15:17:53.139533 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:53.139541 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:53.140159 (+ 618us) server_negotiation.cc:407] Connection header received
0927 15:17:53.140439 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:53.140447 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:53.140673 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:53.140830 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:53.141709 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.143181 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.147867 (+ 4686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.149201 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.149340 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:53.149580 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:53.150121 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 15:17:53.150297 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":246,"threads_started":1}
I20260927 15:17:53.152890 28003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:53.139548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:37771 (local address 127.0.0.1:32778)
0927 15:17:53.144300 (+ 4752us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:53.144318 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:53.144431 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:53.145019 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:53.145024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:53.145046 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:53.145457 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:53.145466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.147267 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.147273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:53.149507 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:53.149519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.152036 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.152043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:53.152263 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:53.152296 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:17:53.152395 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 15:17:53.152525 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4616,"mutex_wait_us":21,"thread_start_us":699,"threads_started":1}
I20260927 15:17:53.152953 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:53.140088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32778 (local address 127.10.33.67:37771)
0927 15:17:53.143908 (+ 3820us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:53.143917 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:53.144557 (+ 640us) server_negotiation.cc:407] Connection header received
0927 15:17:53.144654 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:53.144662 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:53.144847 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:53.145006 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:53.145621 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.147075 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.149651 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.151697 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.151835 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:53.152073 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:53.152460 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 15:17:53.152626 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3634,"thread_start_us":3488,"threads_started":1}
I20260927 15:17:53.153548 27789 tablet_service.cc:1467] Processing CreateTablet for tablet 17b82bad7f8146c1ac173aba5c5cec96 (DEFAULT_TABLE table=test-table [id=fcf387a0e65a4cd2b4aa4c7d04dfb5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:53.154028 27655 tablet_service.cc:1467] Processing CreateTablet for tablet 17b82bad7f8146c1ac173aba5c5cec96 (DEFAULT_TABLE table=test-table [id=fcf387a0e65a4cd2b4aa4c7d04dfb5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:53.155010 27789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17b82bad7f8146c1ac173aba5c5cec96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:53.155371 27655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17b82bad7f8146c1ac173aba5c5cec96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:53.156808 27923 tablet_service.cc:1467] Processing CreateTablet for tablet 17b82bad7f8146c1ac173aba5c5cec96 (DEFAULT_TABLE table=test-table [id=fcf387a0e65a4cd2b4aa4c7d04dfb5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:53.158211 27923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17b82bad7f8146c1ac173aba5c5cec96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:53.166814 28007 tablet_bootstrap.cc:492] T 17b82bad7f8146c1ac173aba5c5cec96 P 069c358ffb33448a973721e31b388822: Bootstrap starting.
I20260927 15:17:53.169526 28008 tablet_bootstrap.cc:492] T 17b82bad7f8146c1ac173aba5c5cec96 P 69612d37448a46fcb422801ee1d1da1d: Bootstrap starting.
I20260927 15:17:53.171705 28007 tablet_bootstrap.cc:654] T 17b82bad7f8146c1ac173aba5c5cec96 P 069c358ffb33448a973721e31b388822: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:53.172452 28008 tablet_bootstrap.cc:654] T 17b82bad7f8146c1ac173aba5c5cec96 P 69612d37448a46fcb422801ee1d1da1d: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:53.173032 28007 log.cc:824] T 17b82bad7f8146c1ac173aba5c5cec96 P 069c358ffb33448a973721e31b388822: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:53.173177 28009 tablet_bootstrap.cc:492] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6: Bootstrap starting.
I20260927 15:17:53.173676 28008 log.cc:824] T 17b82bad7f8146c1ac173aba5c5cec96 P 69612d37448a46fcb422801ee1d1da1d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:53.176348 28009 tablet_bootstrap.cc:654] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:53.176883 28008 tablet_bootstrap.cc:492] T 17b82bad7f8146c1ac173aba5c5cec96 P 69612d37448a46fcb422801ee1d1da1d: No bootstrap required, opened a new log
I20260927 15:17:53.176976 28007 tablet_bootstrap.cc:492] T 17b82bad7f8146c1ac173aba5c5cec96 P 069c358ffb33448a973721e31b388822: No bootstrap required, opened a new log
I20260927 15:17:53.177151 28008 ts_tablet_manager.cc:1397] T 17b82bad7f8146c1ac173aba5c5cec96 P 69612d37448a46fcb422801ee1d1da1d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260927 15:17:53.177187 28007 ts_tablet_manager.cc:1397] T 17b82bad7f8146c1ac173aba5c5cec96 P 069c358ffb33448a973721e31b388822: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260927 15:17:53.177497 28009 log.cc:824] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:53.179960 28009 tablet_bootstrap.cc:492] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6: No bootstrap required, opened a new log
I20260927 15:17:53.180197 28009 ts_tablet_manager.cc:1397] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 15:17:53.187587 28009 raft_consensus.cc:348] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c358ffb33448a973721e31b388822" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37397 } } peers { permanent_uuid: "f76a464efff645e7b6bab60dfd4baee6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 37771 } } peers { permanent_uuid: "69612d37448a46fcb422801ee1d1da1d" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45123 } }
I20260927 15:17:53.187619 28008 raft_consensus.cc:348] T 17b82bad7f8146c1ac173aba5c5cec96 P 69612d37448a46fcb422801ee1d1da1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c358ffb33448a973721e31b388822" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37397 } } peers { permanent_uuid: "f76a464efff645e7b6bab60dfd4baee6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 37771 } } peers { permanent_uuid: "69612d37448a46fcb422801ee1d1da1d" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45123 } }
I20260927 15:17:53.187619 28007 raft_consensus.cc:348] T 17b82bad7f8146c1ac173aba5c5cec96 P 069c358ffb33448a973721e31b388822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c358ffb33448a973721e31b388822" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37397 } } peers { permanent_uuid: "f76a464efff645e7b6bab60dfd4baee6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 37771 } } peers { permanent_uuid: "69612d37448a46fcb422801ee1d1da1d" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45123 } }
I20260927 15:17:53.187963 28009 raft_consensus.cc:374] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:53.188047 28007 raft_consensus.cc:374] T 17b82bad7f8146c1ac173aba5c5cec96 P 069c358ffb33448a973721e31b388822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:53.188062 28008 raft_consensus.cc:374] T 17b82bad7f8146c1ac173aba5c5cec96 P 69612d37448a46fcb422801ee1d1da1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:53.188072 28009 raft_consensus.cc:729] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76a464efff645e7b6bab60dfd4baee6, State: Initialized, Role: FOLLOWER
I20260927 15:17:53.188153 28007 raft_consensus.cc:729] T 17b82bad7f8146c1ac173aba5c5cec96 P 069c358ffb33448a973721e31b388822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 069c358ffb33448a973721e31b388822, State: Initialized, Role: FOLLOWER
I20260927 15:17:53.188153 28008 raft_consensus.cc:729] T 17b82bad7f8146c1ac173aba5c5cec96 P 69612d37448a46fcb422801ee1d1da1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69612d37448a46fcb422801ee1d1da1d, State: Initialized, Role: FOLLOWER
I20260927 15:17:53.188669 28009 consensus_queue.cc:260] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c358ffb33448a973721e31b388822" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37397 } } peers { permanent_uuid: "f76a464efff645e7b6bab60dfd4baee6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 37771 } } peers { permanent_uuid: "69612d37448a46fcb422801ee1d1da1d" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45123 } }
I20260927 15:17:53.188815 28008 consensus_queue.cc:260] T 17b82bad7f8146c1ac173aba5c5cec96 P 69612d37448a46fcb422801ee1d1da1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c358ffb33448a973721e31b388822" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37397 } } peers { permanent_uuid: "f76a464efff645e7b6bab60dfd4baee6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 37771 } } peers { permanent_uuid: "69612d37448a46fcb422801ee1d1da1d" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45123 } }
I20260927 15:17:53.188813 28007 consensus_queue.cc:260] T 17b82bad7f8146c1ac173aba5c5cec96 P 069c358ffb33448a973721e31b388822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c358ffb33448a973721e31b388822" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37397 } } peers { permanent_uuid: "f76a464efff645e7b6bab60dfd4baee6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 37771 } } peers { permanent_uuid: "69612d37448a46fcb422801ee1d1da1d" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45123 } }
I20260927 15:17:53.189802 27988 heartbeater.cc:503] Master 127.10.33.126:45085 was elected leader, sending a full tablet report...
I20260927 15:17:53.190498 28009 ts_tablet_manager.cc:1428] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260927 15:17:53.191888 27854 heartbeater.cc:503] Master 127.10.33.126:45085 was elected leader, sending a full tablet report...
I20260927 15:17:53.191943 28008 ts_tablet_manager.cc:1428] T 17b82bad7f8146c1ac173aba5c5cec96 P 69612d37448a46fcb422801ee1d1da1d: Time spent starting tablet: real 0.015s user 0.008s sys 0.004s
I20260927 15:17:53.192391 28007 ts_tablet_manager.cc:1428] T 17b82bad7f8146c1ac173aba5c5cec96 P 069c358ffb33448a973721e31b388822: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260927 15:17:53.253908 27721 tablet.cc:2369] T 17b82bad7f8146c1ac173aba5c5cec96 P 69612d37448a46fcb422801ee1d1da1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:17:53.262662 28013 raft_consensus.cc:482] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:17:53.262951 28013 raft_consensus.cc:504] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c358ffb33448a973721e31b388822" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37397 } } peers { permanent_uuid: "f76a464efff645e7b6bab60dfd4baee6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 37771 } } peers { permanent_uuid: "69612d37448a46fcb422801ee1d1da1d" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45123 } }
I20260927 15:17:53.264813 28013 leader_election.cc:288] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 069c358ffb33448a973721e31b388822 (127.10.33.66:37397), 69612d37448a46fcb422801ee1d1da1d (127.10.33.65:45123)
I20260927 15:17:53.270545 27991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:53.265156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:45123 (local address 127.10.33.67:44765)
0927 15:17:53.265527 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:53.265548 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:53.265711 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:53.266123 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:53.266129 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:53.266156 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:53.266347 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:53.266355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.267348 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.267352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:53.268656 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:53.268668 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.269757 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.269762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:53.269965 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:53.270007 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 15:17:53.270098 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 15:17:53.270241 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"spinlock_wait_cycles":20736}
I20260927 15:17:53.270581 28004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:53.265416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:44765 (local address 127.10.33.65:45123)
0927 15:17:53.265620 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:53.265626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:53.265644 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:17:53.265910 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:53.265915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:53.265976 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:53.266062 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:53.266485 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.267212 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.268816 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.269606 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.269754 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:53.269832 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:53.270169 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 15:17:53.270325 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 15:17:53.271273 28001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:53.265338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:42905 (local address 127.10.33.66:37397)
0927 15:17:53.265553 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:53.265559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:53.266210 (+ 651us) server_negotiation.cc:407] Connection header received
0927 15:17:53.266399 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:53.266403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:53.266463 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:53.266543 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:53.267043 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.267728 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.269353 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.270221 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.270369 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:53.270438 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:53.270903 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 15:17:53.271032 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:17:53.271371 28016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:53.265261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:37397 (local address 127.10.33.67:42905)
0927 15:17:53.266106 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:53.266120 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:53.266272 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:53.266573 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:53.266578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:53.266601 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:53.266897 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:53.266906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.267869 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.267872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:53.269208 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:53.269219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.270581 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.270587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:53.270744 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:53.270779 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 15:17:53.270860 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 15:17:53.270999 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"spinlock_wait_cycles":22016,"thread_start_us":487,"threads_started":1}
I20260927 15:17:53.272056 27675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b82bad7f8146c1ac173aba5c5cec96" candidate_uuid: "f76a464efff645e7b6bab60dfd4baee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69612d37448a46fcb422801ee1d1da1d" is_pre_election: true
I20260927 15:17:53.272264 27809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b82bad7f8146c1ac173aba5c5cec96" candidate_uuid: "f76a464efff645e7b6bab60dfd4baee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "069c358ffb33448a973721e31b388822" is_pre_election: true
I20260927 15:17:53.272672 27675 raft_consensus.cc:2454] T 17b82bad7f8146c1ac173aba5c5cec96 P 69612d37448a46fcb422801ee1d1da1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f76a464efff645e7b6bab60dfd4baee6 in term 0.
I20260927 15:17:53.272725 27809 raft_consensus.cc:2454] T 17b82bad7f8146c1ac173aba5c5cec96 P 069c358ffb33448a973721e31b388822 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f76a464efff645e7b6bab60dfd4baee6 in term 0.
I20260927 15:17:53.273433 27878 leader_election.cc:302] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 069c358ffb33448a973721e31b388822, 69612d37448a46fcb422801ee1d1da1d, f76a464efff645e7b6bab60dfd4baee6; no voters:
I20260927 15:17:53.273824 28013 raft_consensus.cc:2781] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:17:53.274000 28013 raft_consensus.cc:482] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:17:53.274103 28013 raft_consensus.cc:3037] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:53.276230 28013 raft_consensus.cc:504] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c358ffb33448a973721e31b388822" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37397 } } peers { permanent_uuid: "f76a464efff645e7b6bab60dfd4baee6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 37771 } } peers { permanent_uuid: "69612d37448a46fcb422801ee1d1da1d" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45123 } }
I20260927 15:17:53.276880 28013 leader_election.cc:288] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6 [CANDIDATE]: Term 1 election: Requested vote from peers 069c358ffb33448a973721e31b388822 (127.10.33.66:37397), 69612d37448a46fcb422801ee1d1da1d (127.10.33.65:45123)
I20260927 15:17:53.277341 27809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b82bad7f8146c1ac173aba5c5cec96" candidate_uuid: "f76a464efff645e7b6bab60dfd4baee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "069c358ffb33448a973721e31b388822"
I20260927 15:17:53.277410 27675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b82bad7f8146c1ac173aba5c5cec96" candidate_uuid: "f76a464efff645e7b6bab60dfd4baee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69612d37448a46fcb422801ee1d1da1d"
I20260927 15:17:53.277532 27809 raft_consensus.cc:3037] T 17b82bad7f8146c1ac173aba5c5cec96 P 069c358ffb33448a973721e31b388822 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:53.277624 27675 raft_consensus.cc:3037] T 17b82bad7f8146c1ac173aba5c5cec96 P 69612d37448a46fcb422801ee1d1da1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:53.279794 27809 raft_consensus.cc:2454] T 17b82bad7f8146c1ac173aba5c5cec96 P 069c358ffb33448a973721e31b388822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f76a464efff645e7b6bab60dfd4baee6 in term 1.
I20260927 15:17:53.279935 27675 raft_consensus.cc:2454] T 17b82bad7f8146c1ac173aba5c5cec96 P 69612d37448a46fcb422801ee1d1da1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f76a464efff645e7b6bab60dfd4baee6 in term 1.
I20260927 15:17:53.280231 27878 leader_election.cc:302] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 069c358ffb33448a973721e31b388822, f76a464efff645e7b6bab60dfd4baee6; no voters:
I20260927 15:17:53.280575 28013 raft_consensus.cc:2781] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:17:53.281011 28013 raft_consensus.cc:686] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6 [term 1 LEADER]: Becoming Leader. State: Replica: f76a464efff645e7b6bab60dfd4baee6, State: Running, Role: LEADER
I20260927 15:17:53.281584 28013 consensus_queue.cc:237] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c358ffb33448a973721e31b388822" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37397 } } peers { permanent_uuid: "f76a464efff645e7b6bab60dfd4baee6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 37771 } } peers { permanent_uuid: "69612d37448a46fcb422801ee1d1da1d" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45123 } }
I20260927 15:17:53.286374 27531 catalog_manager.cc:5495] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f76a464efff645e7b6bab60dfd4baee6 (127.10.33.67). New cstate: current_term: 1 leader_uuid: "f76a464efff645e7b6bab60dfd4baee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c358ffb33448a973721e31b388822" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f76a464efff645e7b6bab60dfd4baee6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 37771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69612d37448a46fcb422801ee1d1da1d" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45123 } health_report { overall_health: UNKNOWN } } }
W20260927 15:17:53.288231 27855 tablet.cc:2369] T 17b82bad7f8146c1ac173aba5c5cec96 P 069c358ffb33448a973721e31b388822: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:17:53.308740 27989 tablet.cc:2369] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:17:53.314494 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:53.309099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32780 (local address 127.10.33.67:37771)
0927 15:17:53.309310 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:53.309316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:53.309336 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:17:53.309467 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:53.309471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:53.309557 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:53.309691 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:17:53.310099 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.310817 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.312304 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.312712 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.312833 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:53.313353 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:17:53.313907 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:17:53.314037 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:53.314139 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 15:17:53.314226 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:17:53.322644 27809 raft_consensus.cc:1261] T 17b82bad7f8146c1ac173aba5c5cec96 P 069c358ffb33448a973721e31b388822 [term 1 FOLLOWER]: Refusing update from remote peer f76a464efff645e7b6bab60dfd4baee6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:17:53.322650 27675 raft_consensus.cc:1261] T 17b82bad7f8146c1ac173aba5c5cec96 P 69612d37448a46fcb422801ee1d1da1d [term 1 FOLLOWER]: Refusing update from remote peer f76a464efff645e7b6bab60dfd4baee6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:17:53.323590 28017 consensus_queue.cc:1035] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69612d37448a46fcb422801ee1d1da1d" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 45123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:17:53.323966 28013 consensus_queue.cc:1035] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "069c358ffb33448a973721e31b388822" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:17:53.333664 28020 mvcc.cc:205] Tried to move back new op lower bound from 7333979231520030720 to 7333979231366176768. Current Snapshot: MvccSnapshot[applied={T|T < 7333979231520030720}]
I20260927 15:17:53.337432 28022 mvcc.cc:205] Tried to move back new op lower bound from 7333979231520030720 to 7333979231366176768. Current Snapshot: MvccSnapshot[applied={T|T < 7333979231520030720 or (T in {7333979231520030720})}]
I20260927 15:17:53.350353 27531 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:17:53.350760 27531 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:17:53.355022 27531 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e298c1e98c94d5a97ea772914b3a473: Sending DeleteTablet for 3 replicas of tablet 17b82bad7f8146c1ac173aba5c5cec96
I20260927 15:17:53.356171 27655 tablet_service.cc:1514] Processing DeleteTablet for tablet 17b82bad7f8146c1ac173aba5c5cec96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:53 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:38064
I20260927 15:17:53.356164 27789 tablet_service.cc:1514] Processing DeleteTablet for tablet 17b82bad7f8146c1ac173aba5c5cec96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:53 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:46894
I20260927 15:17:53.357015 28030 tablet_replica.cc:330] T 17b82bad7f8146c1ac173aba5c5cec96 P 69612d37448a46fcb422801ee1d1da1d: stopping tablet replica
I20260927 15:17:53.357080 27923 tablet_service.cc:1514] Processing DeleteTablet for tablet 17b82bad7f8146c1ac173aba5c5cec96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:53 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:32778
I20260927 15:17:53.357504 28030 raft_consensus.cc:2229] T 17b82bad7f8146c1ac173aba5c5cec96 P 69612d37448a46fcb422801ee1d1da1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:17:53.358136 28030 raft_consensus.cc:2258] T 17b82bad7f8146c1ac173aba5c5cec96 P 69612d37448a46fcb422801ee1d1da1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:53.360157 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 27592
I20260927 15:17:53.360463 28030 ts_tablet_manager.cc:1905] T 17b82bad7f8146c1ac173aba5c5cec96 P 69612d37448a46fcb422801ee1d1da1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:53.368696 28030 ts_tablet_manager.cc:1918] T 17b82bad7f8146c1ac173aba5c5cec96 P 69612d37448a46fcb422801ee1d1da1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:53.368949 28030 log.cc:1196] T 17b82bad7f8146c1ac173aba5c5cec96 P 69612d37448a46fcb422801ee1d1da1d: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/17b82bad7f8146c1ac173aba5c5cec96
I20260927 15:17:53.369531 28030 ts_tablet_manager.cc:1939] T 17b82bad7f8146c1ac173aba5c5cec96 P 69612d37448a46fcb422801ee1d1da1d: Deleting consensus metadata
I20260927 15:17:53.371582 27518 catalog_manager.cc:4841] TS 69612d37448a46fcb422801ee1d1da1d (127.10.33.65:45123): tablet 17b82bad7f8146c1ac173aba5c5cec96 (table test-table [id=fcf387a0e65a4cd2b4aa4c7d04dfb5d7]) successfully deleted
I20260927 15:17:53.375070 28032 tablet_replica.cc:330] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6: stopping tablet replica
I20260927 15:17:53.375711 28032 raft_consensus.cc:2229] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:17:53.376477 28032 raft_consensus.cc:2258] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:53.378919 28032 ts_tablet_manager.cc:1905] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 15:17:53 dist-test-slave-f5h6 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.65@KRBTEST.COM
I20260927 15:17:53.386212 28032 ts_tablet_manager.cc:1918] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:53.386934 28031 tablet_replica.cc:330] T 17b82bad7f8146c1ac173aba5c5cec96 P 069c358ffb33448a973721e31b388822: stopping tablet replica
I20260927 15:17:53.387432 28031 raft_consensus.cc:2229] T 17b82bad7f8146c1ac173aba5c5cec96 P 069c358ffb33448a973721e31b388822 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:17:53.387984 28031 raft_consensus.cc:2258] T 17b82bad7f8146c1ac173aba5c5cec96 P 069c358ffb33448a973721e31b388822 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:53.388437 28032 log.cc:1196] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/17b82bad7f8146c1ac173aba5c5cec96
I20260927 15:17:53.389523 28032 ts_tablet_manager.cc:1939] T 17b82bad7f8146c1ac173aba5c5cec96 P f76a464efff645e7b6bab60dfd4baee6: Deleting consensus metadata
I20260927 15:17:53.390097 28031 ts_tablet_manager.cc:1905] T 17b82bad7f8146c1ac173aba5c5cec96 P 069c358ffb33448a973721e31b388822: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:53.391450 27517 catalog_manager.cc:4841] TS f76a464efff645e7b6bab60dfd4baee6 (127.10.33.67:37771): tablet 17b82bad7f8146c1ac173aba5c5cec96 (table test-table [id=fcf387a0e65a4cd2b4aa4c7d04dfb5d7]) successfully deleted
I20260927 15:17:53.394158 28004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:53.361951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38074 (local address 127.10.33.65:45123)
0927 15:17:53.362130 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:53.362140 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:53.374424 (+ 12284us) server_negotiation.cc:407] Connection header received
0927 15:17:53.374632 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:53.374637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:53.374732 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:53.374821 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:53.376662 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.377710 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.379620 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.380222 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.380320 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:53.386154 (+ 5834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:53.386175 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:53.386196 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:53.386247 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:53.390084 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:53.391124 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:53.391134 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:53.391137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:53.391524 (+ 387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:53.392024 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:53.392029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:53.392032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:53.392534 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:53.392825 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:53.393489 (+ 664us) server_negotiation.cc:299] Negotiation successful
0927 15:17:53.393741 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:17:53.396150 27715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:17:53.397212 28031 ts_tablet_manager.cc:1918] T 17b82bad7f8146c1ac173aba5c5cec96 P 069c358ffb33448a973721e31b388822: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:53.397488 28031 log.cc:1196] T 17b82bad7f8146c1ac173aba5c5cec96 P 069c358ffb33448a973721e31b388822: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/17b82bad7f8146c1ac173aba5c5cec96
I20260927 15:17:53.398202 28031 ts_tablet_manager.cc:1939] T 17b82bad7f8146c1ac173aba5c5cec96 P 069c358ffb33448a973721e31b388822: Deleting consensus metadata
I20260927 15:17:53.400152 27517 catalog_manager.cc:4841] TS 069c358ffb33448a973721e31b388822 (127.10.33.66:37397): tablet 17b82bad7f8146c1ac173aba5c5cec96 (table test-table [id=fcf387a0e65a4cd2b4aa4c7d04dfb5d7]) successfully deleted
I20260927 15:17:53.488687 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 27592
I20260927 15:17:53.510862 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 27726
Sep 27 15:17:53 dist-test-slave-f5h6 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.66@KRBTEST.COM
I20260927 15:17:53.530179 28001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:53.511714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46904 (local address 127.10.33.66:37397)
0927 15:17:53.511878 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:53.511882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:53.511900 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:17:53.511994 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:53.511998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:53.512071 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:53.512152 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:53.513835 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.514664 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.516784 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.517418 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.517574 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:53.523342 (+ 5768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:53.523361 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:53.523380 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:53.523426 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:53.526694 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:53.527752 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:53.527762 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:53.527764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:53.527997 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:53.528437 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:53.528441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:53.528443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:53.528954 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:53.529181 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:53.529605 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 15:17:53.529795 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:17:53.530946 27849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:17:53.619920 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 27726
I20260927 15:17:53.634325 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 27860
Sep 27 15:17:53 dist-test-slave-f5h6 krb5kdc[27486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.67@KRBTEST.COM
I20260927 15:17:53.651386 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:53.635144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32792 (local address 127.10.33.67:37771)
0927 15:17:53.635354 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:53.635361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:53.635383 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:17:53.635450 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:53.635456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:53.635546 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:53.635669 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:53.637011 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.638303 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.639591 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:53.640319 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:53.640465 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:53.645087 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:53.645102 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:53.645119 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:53.645174 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:53.648154 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:53.649114 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:53.649121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:53.649123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:53.649284 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:53.649725 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:53.649730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:53.649733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:53.650164 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:53.650401 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:53.650768 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 15:17:53.651010 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 15:17:53.652232 27983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:17:53.775808 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 27860
I20260927 15:17:53.792737 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 27499
I20260927 15:17:53.794094 27561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:17:53.905616 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 27499
2026-09-27T15:17:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4256 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T15:17:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:17:53Z Disabled control of system clock
I20260927 15:17:53.980490 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:41611
--webserver_interface=127.10.33.126
--webserver_port=0
--builtin_ntp_servers=127.10.33.84:45203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:41611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:17:54.314108 28041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:54.314422 28041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:54.314507 28041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:54.321906 28041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:17:54.322034 28041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:54.322072 28041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:54.322129 28041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:17:54.322172 28041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:17:54.331787 28041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:45203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:41611
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:41611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:17:54.333235 28041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:54.335007 28041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:54.346868 28046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:54.346930 28049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:54.346912 28047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:54.347769 28041 server_base.cc:1034] running on GCE node
I20260927 15:17:54.348592 28041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:54.350095 28041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:54.351318 28041 hybrid_clock.cc:648] HybridClock initialized: now 1790522274351269 us; error 39 us; skew 500 ppm
I20260927 15:17:54.351742 28041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:17:54.360934 28041 webserver.cc:460] Webserver started at http://127.10.33.126:45801/ using document root <none> and password file <none>
I20260927 15:17:54.361842 28041 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:54.362031 28041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:54.362383 28041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:54.365375 28041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "de289cb092ba402aaf0434a1b78cd625"
format_stamp: "Formatted at 2026-09-27 15:17:54 on dist-test-slave-f5h6"
server_key: "9285e4f743e41efb02fb50538885cc4e"
server_key_iv: "d37c5db369573e42e7a83cdee5f99f07"
server_key_version: "encryptionkey@0"
I20260927 15:17:54.366251 28041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "de289cb092ba402aaf0434a1b78cd625"
format_stamp: "Formatted at 2026-09-27 15:17:54 on dist-test-slave-f5h6"
server_key: "9285e4f743e41efb02fb50538885cc4e"
server_key_iv: "d37c5db369573e42e7a83cdee5f99f07"
server_key_version: "encryptionkey@0"
I20260927 15:17:54.371479 28041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:17:54.375268 28055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:54.376696 28041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:17:54.376873 28041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "de289cb092ba402aaf0434a1b78cd625"
format_stamp: "Formatted at 2026-09-27 15:17:54 on dist-test-slave-f5h6"
server_key: "9285e4f743e41efb02fb50538885cc4e"
server_key_iv: "d37c5db369573e42e7a83cdee5f99f07"
server_key_version: "encryptionkey@0"
I20260927 15:17:54.377092 28041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:54.389716 28041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:54.390587 28041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:54.390846 28041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:54.408547 28041 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:41611
I20260927 15:17:54.408577 28106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:41611 every 8 connection(s)
I20260927 15:17:54.410053 28041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:17:54.413810 28107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:54.414646 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 28041
I20260927 15:17:54.414801 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:17:54.415256 10373 external_mini_cluster.cc:1428] Setting key b8afcedd69ce34d128d17a79a2afe664
I20260927 15:17:54.428244 28107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625: Bootstrap starting.
I20260927 15:17:54.431018 28110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:54.417285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41650 (local address 127.10.33.126:41611)
0927 15:17:54.418582 (+ 1297us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:54.418603 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:54.418660 (+ 57us) server_negotiation.cc:407] Connection header received
0927 15:17:54.420390 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:54.420428 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:54.421111 (+ 683us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:54.421602 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:54.422173 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:54.424070 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:54.425593 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:54.426669 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:54.426878 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:54.428047 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:54.428059 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:54.428079 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:17:54.428816 (+ 737us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:54.428927 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:17:54.428955 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:54.429085 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:54.429298 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 15:17:54.429638 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":426,"threads_started":1}
I20260927 15:17:54.432427 28107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:54.433557 28107 log.cc:824] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:54.437117 28107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625: No bootstrap required, opened a new log
I20260927 15:17:54.445715 28107 raft_consensus.cc:348] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de289cb092ba402aaf0434a1b78cd625" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 41611 } }
I20260927 15:17:54.446193 28107 raft_consensus.cc:374] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:54.446318 28107 raft_consensus.cc:729] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de289cb092ba402aaf0434a1b78cd625, State: Initialized, Role: FOLLOWER
I20260927 15:17:54.446947 28107 consensus_queue.cc:260] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de289cb092ba402aaf0434a1b78cd625" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 41611 } }
I20260927 15:17:54.447199 28107 raft_consensus.cc:388] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:17:54.447356 28107 raft_consensus.cc:482] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:17:54.447536 28107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:54.450045 28107 raft_consensus.cc:504] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de289cb092ba402aaf0434a1b78cd625" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 41611 } }
I20260927 15:17:54.450623 28107 leader_election.cc:302] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de289cb092ba402aaf0434a1b78cd625; no voters:
I20260927 15:17:54.451304 28107 leader_election.cc:288] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:17:54.451407 28112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:17:54.452696 28112 raft_consensus.cc:686] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625 [term 1 LEADER]: Becoming Leader. State: Replica: de289cb092ba402aaf0434a1b78cd625, State: Running, Role: LEADER
I20260927 15:17:54.453218 28112 consensus_queue.cc:237] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de289cb092ba402aaf0434a1b78cd625" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 41611 } }
I20260927 15:17:54.453733 28107 sys_catalog.cc:564] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:17:54.457049 28113 sys_catalog.cc:455] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de289cb092ba402aaf0434a1b78cd625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de289cb092ba402aaf0434a1b78cd625" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 41611 } } }
I20260927 15:17:54.457690 28113 sys_catalog.cc:458] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625 [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:54.457593 28114 sys_catalog.cc:455] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de289cb092ba402aaf0434a1b78cd625. Latest consensus state: current_term: 1 leader_uuid: "de289cb092ba402aaf0434a1b78cd625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de289cb092ba402aaf0434a1b78cd625" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 41611 } } }
I20260927 15:17:54.457932 28114 sys_catalog.cc:458] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625 [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:54.458699 28122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:17:54.463924 28122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:17:54.472832 28122 catalog_manager.cc:1347] Generated new cluster ID: 359f266edbae4ff8b0a13b64e063204d
I20260927 15:17:54.473019 28122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:17:54.501374 28122 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:17:54.502777 28122 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:17:54.511418 28122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625: Generated new TSK 0
I20260927 15:17:54.512580 28122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:17:54.525777 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:41611
--builtin_ntp_servers=127.10.33.84:45203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:17:54.865193 28131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:54.865507 28131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:54.865631 28131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:54.873123 28131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:54.873258 28131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:54.873397 28131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:17:54.883339 28131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:45203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:41611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:17:54.884676 28131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:54.886490 28131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:54.900197 28136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:54.900431 28137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:54.900478 28139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:54.901087 28131 server_base.cc:1034] running on GCE node
I20260927 15:17:54.901814 28131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:54.902920 28131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:54.904145 28131 hybrid_clock.cc:648] HybridClock initialized: now 1790522274904090 us; error 57 us; skew 500 ppm
I20260927 15:17:54.904485 28131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:17:54.906839 28131 webserver.cc:460] Webserver started at http://127.10.33.65:37247/ using document root <none> and password file <none>
I20260927 15:17:54.907527 28131 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:54.907630 28131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:54.907967 28131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:54.910964 28131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "ac28e997bca34402b439c07b60cb1370"
format_stamp: "Formatted at 2026-09-27 15:17:54 on dist-test-slave-f5h6"
server_key: "634e8b26bad5d59824c70a0166388a52"
server_key_iv: "d2eb12fb10415ea755d04011553a597f"
server_key_version: "encryptionkey@0"
I20260927 15:17:54.911769 28131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "ac28e997bca34402b439c07b60cb1370"
format_stamp: "Formatted at 2026-09-27 15:17:54 on dist-test-slave-f5h6"
server_key: "634e8b26bad5d59824c70a0166388a52"
server_key_iv: "d2eb12fb10415ea755d04011553a597f"
server_key_version: "encryptionkey@0"
I20260927 15:17:54.917073 28131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:17:54.920855 28145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:54.922519 28131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:17:54.922705 28131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "ac28e997bca34402b439c07b60cb1370"
format_stamp: "Formatted at 2026-09-27 15:17:54 on dist-test-slave-f5h6"
server_key: "634e8b26bad5d59824c70a0166388a52"
server_key_iv: "d2eb12fb10415ea755d04011553a597f"
server_key_version: "encryptionkey@0"
I20260927 15:17:54.922948 28131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:54.949414 28131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:54.950418 28131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:54.950735 28131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:54.951813 28131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:54.953387 28131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:54.953491 28131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:54.953604 28131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:54.953685 28131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:54.986461 28131 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:33683
I20260927 15:17:54.986536 28257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:33683 every 8 connection(s)
I20260927 15:17:54.988041 28131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:17:54.989616 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 28131
I20260927 15:17:54.989794 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:17:54.990397 10373 external_mini_cluster.cc:1428] Setting key 4964a10c90ffffb20eed202b4c12a078
I20260927 15:17:54.993611 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:41611
--builtin_ntp_servers=127.10.33.84:45203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:17:55.003238 28260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:54.990947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:56501 (local address 127.10.33.126:41611)
0927 15:17:54.991668 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:54.991673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:54.992894 (+ 1221us) server_negotiation.cc:407] Connection header received
0927 15:17:54.995412 (+ 2518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:54.995417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:54.995497 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:54.995597 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:54.996953 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:54.997741 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:55.000021 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:55.000509 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:55.000633 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:55.002276 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:55.002286 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:55.002300 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:17:55.002331 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:55.002388 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:17:55.002407 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:55.002516 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:55.002717 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 15:17:55.002841 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":523,"threads_started":1}
I20260927 15:17:55.004398 28261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:54.991612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:41611 (local address 127.10.33.65:56501)
0927 15:17:54.992707 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:54.992764 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:54.995086 (+ 2322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:54.995840 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:54.995852 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:54.995987 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:54.996746 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:54.996768 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:54.997988 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:54.998006 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:54.999832 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:54.999849 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:55.000754 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:55.000760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:55.001056 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:55.001852 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:17:55.001885 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:55.001954 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:17:55.002051 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:55.002503 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:55.002509 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:55.002514 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:17:55.002653 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 15:17:55.003041 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":351,"threads_started":1}
I20260927 15:17:55.006695 28258 heartbeater.cc:347] Connected to a master server at 127.10.33.126:41611
I20260927 15:17:55.007014 28258 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:55.007763 28258 heartbeater.cc:511] Master 127.10.33.126:41611 requested a full tablet report, sending...
I20260927 15:17:55.010015 28072 ts_manager.cc:194] Registered new tserver with Master: ac28e997bca34402b439c07b60cb1370 (127.10.33.65:33683)
I20260927 15:17:55.011677 28072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.33.65:56501
W20260927 15:17:55.331935 28262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:55.332216 28262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:55.332326 28262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:55.339536 28262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:55.339650 28262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:55.339785 28262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:17:55.349108 28262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:45203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:41611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:17:55.350410 28262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:55.352172 28262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:55.365557 28267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:55.365449 28268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:55.365477 28270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:55.366230 28262 server_base.cc:1034] running on GCE node
I20260927 15:17:55.366869 28262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:55.367843 28262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:55.369073 28262 hybrid_clock.cc:648] HybridClock initialized: now 1790522275369026 us; error 46 us; skew 500 ppm
I20260927 15:17:55.369398 28262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:17:55.371829 28262 webserver.cc:460] Webserver started at http://127.10.33.66:39795/ using document root <none> and password file <none>
I20260927 15:17:55.372529 28262 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:55.372632 28262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:55.372927 28262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:55.376225 28262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "274d568421d0469081781e2b71e30f5f"
format_stamp: "Formatted at 2026-09-27 15:17:55 on dist-test-slave-f5h6"
server_key: "fd792e05cfc80eb1448a745795112a9f"
server_key_iv: "c6b5919db250c6cb03093abc8cbbba08"
server_key_version: "encryptionkey@0"
I20260927 15:17:55.377100 28262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "274d568421d0469081781e2b71e30f5f"
format_stamp: "Formatted at 2026-09-27 15:17:55 on dist-test-slave-f5h6"
server_key: "fd792e05cfc80eb1448a745795112a9f"
server_key_iv: "c6b5919db250c6cb03093abc8cbbba08"
server_key_version: "encryptionkey@0"
I20260927 15:17:55.382715 28262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 15:17:55.386312 28276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:55.387809 28262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 15:17:55.387980 28262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "274d568421d0469081781e2b71e30f5f"
format_stamp: "Formatted at 2026-09-27 15:17:55 on dist-test-slave-f5h6"
server_key: "fd792e05cfc80eb1448a745795112a9f"
server_key_iv: "c6b5919db250c6cb03093abc8cbbba08"
server_key_version: "encryptionkey@0"
I20260927 15:17:55.388192 28262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:55.448129 28262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:55.449067 28262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:55.449366 28262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:55.450434 28262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:55.452025 28262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:55.452124 28262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:55.452242 28262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:55.452311 28262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:55.483959 28262 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:36161
I20260927 15:17:55.484050 28388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:36161 every 8 connection(s)
I20260927 15:17:55.485288 28262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:17:55.493531 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 28262
I20260927 15:17:55.493692 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:17:55.494148 10373 external_mini_cluster.cc:1428] Setting key d753042fe5e2249b6ea05e7dbf3b00b5
I20260927 15:17:55.497619 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:41611
--builtin_ntp_servers=127.10.33.84:45203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:17:55.499568 28260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:55.488049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:45413 (local address 127.10.33.126:41611)
0927 15:17:55.488222 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:55.488227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:55.489734 (+ 1507us) server_negotiation.cc:407] Connection header received
0927 15:17:55.491622 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:55.491627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:55.491710 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:55.491822 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:55.493162 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:55.494084 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:55.496138 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:55.496678 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:55.496790 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:55.498603 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:55.498610 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:55.498613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:17:55.498634 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:55.498670 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:17:55.498684 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:55.498758 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:55.499026 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 15:17:55.499167 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:17:55.501044 28391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:55.488502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:41611 (local address 127.10.33.66:45413)
0927 15:17:55.489557 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:55.489599 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:55.491291 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:55.492087 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:55.492096 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:55.492200 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:55.492931 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:55.492956 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:55.494251 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:55.494257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:55.495870 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:55.495884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:55.496951 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:55.496959 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:55.497304 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:55.498149 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:17:55.498154 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:55.498223 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:17:55.498340 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:55.498779 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:55.498789 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:55.498797 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 15:17:55.499022 (+ 225us) client_negotiation.cc:240] Negotiation successful
0927 15:17:55.499489 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":408,"threads_started":1}
I20260927 15:17:55.502651 28389 heartbeater.cc:347] Connected to a master server at 127.10.33.126:41611
I20260927 15:17:55.503008 28389 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:55.503974 28389 heartbeater.cc:511] Master 127.10.33.126:41611 requested a full tablet report, sending...
I20260927 15:17:55.505800 28072 ts_manager.cc:194] Registered new tserver with Master: 274d568421d0469081781e2b71e30f5f (127.10.33.66:36161)
I20260927 15:17:55.506903 28072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.33.66:45413
W20260927 15:17:55.824435 28392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:55.824728 28392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:55.824827 28392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:55.832058 28392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:55.832168 28392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:55.832275 28392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:17:55.841437 28392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:45203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:41611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:17:55.842730 28392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:55.844425 28392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:55.857169 28398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:55.857191 28400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:55.857731 28397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:55.863619 28392 server_base.cc:1034] running on GCE node
I20260927 15:17:55.864285 28392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:55.865273 28392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:55.866468 28392 hybrid_clock.cc:648] HybridClock initialized: now 1790522275866393 us; error 64 us; skew 500 ppm
I20260927 15:17:55.866775 28392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:17:55.868999 28392 webserver.cc:460] Webserver started at http://127.10.33.67:37797/ using document root <none> and password file <none>
I20260927 15:17:55.869656 28392 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:55.869748 28392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:55.870102 28392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:55.872908 28392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "b150238ef3524ce48001dfe78dd2abf6"
format_stamp: "Formatted at 2026-09-27 15:17:55 on dist-test-slave-f5h6"
server_key: "0e77e736b6b0a6c65e04a865d6d9dd01"
server_key_iv: "1298fb6853aec0ac4397637c9cc16bfd"
server_key_version: "encryptionkey@0"
I20260927 15:17:55.873677 28392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "b150238ef3524ce48001dfe78dd2abf6"
format_stamp: "Formatted at 2026-09-27 15:17:55 on dist-test-slave-f5h6"
server_key: "0e77e736b6b0a6c65e04a865d6d9dd01"
server_key_iv: "1298fb6853aec0ac4397637c9cc16bfd"
server_key_version: "encryptionkey@0"
I20260927 15:17:55.879158 28392 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 15:17:55.882812 28406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:55.884183 28392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:17:55.884326 28392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "b150238ef3524ce48001dfe78dd2abf6"
format_stamp: "Formatted at 2026-09-27 15:17:55 on dist-test-slave-f5h6"
server_key: "0e77e736b6b0a6c65e04a865d6d9dd01"
server_key_iv: "1298fb6853aec0ac4397637c9cc16bfd"
server_key_version: "encryptionkey@0"
I20260927 15:17:55.884537 28392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:55.902105 28392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:55.902946 28392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:55.903225 28392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:55.904214 28392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:55.905732 28392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:55.905823 28392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:55.905965 28392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:55.906033 28392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:55.936583 28392 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:35247
I20260927 15:17:55.936655 28518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:35247 every 8 connection(s)
I20260927 15:17:55.938053 28392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:17:55.944586 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 28392
I20260927 15:17:55.944767 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:17:55.945290 10373 external_mini_cluster.cc:1428] Setting key 245dcd1c9c9a8cec742e824ffcf3f72b
I20260927 15:17:55.951421 28260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:55.940784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:46097 (local address 127.10.33.126:41611)
0927 15:17:55.941030 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:55.941050 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:55.942499 (+ 1449us) server_negotiation.cc:407] Connection header received
0927 15:17:55.944376 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:55.944381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:55.944466 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:55.944605 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:55.946093 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:55.946873 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:55.948479 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:55.948899 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:55.949009 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:55.950605 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:55.950611 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:55.950614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:17:55.950633 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:55.950665 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:17:55.950677 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:55.950754 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:55.950988 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 15:17:55.951095 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 15:17:55.952458 28521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:55.941323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:41611 (local address 127.10.33.67:46097)
0927 15:17:55.942328 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:55.942371 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:55.944052 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:55.944875 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:55.944887 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:55.945025 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:55.945851 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:55.945921 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:55.947031 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:55.947035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:55.948329 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:55.948341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:55.949128 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:55.949135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:55.949363 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:55.950230 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:17:55.950232 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:55.950278 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:17:55.950374 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:55.950771 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:55.950782 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:55.950790 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 15:17:55.950948 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 15:17:55.951436 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":310,"threads_started":1}
I20260927 15:17:55.953850 28519 heartbeater.cc:347] Connected to a master server at 127.10.33.126:41611
I20260927 15:17:55.954257 28519 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:55.955049 28519 heartbeater.cc:511] Master 127.10.33.126:41611 requested a full tablet report, sending...
I20260927 15:17:55.956689 28072 ts_manager.cc:194] Registered new tserver with Master: b150238ef3524ce48001dfe78dd2abf6 (127.10.33.67:35247)
I20260927 15:17:55.957567 28072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.33.67:46097
I20260927 15:17:55.961197 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:17:55.971784 28260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:55.965320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41654 (local address 127.10.33.126:41611)
0927 15:17:55.965486 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:55.965491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:55.965974 (+ 483us) server_negotiation.cc:407] Connection header received
0927 15:17:55.966169 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:55.966174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:55.966241 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:55.966334 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:55.966908 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:55.967673 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:55.969198 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:55.969628 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:55.969730 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:55.970990 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:55.970996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:55.971000 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:17:55.971022 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:55.971058 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:17:55.971072 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:55.971153 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:55.971365 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 15:17:55.971466 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:17:55.976466 28072 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:17:55.983587 28072 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:17:56.012755 28531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:56.000122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33683 (local address 127.0.0.1:37282)
0927 15:17:56.002440 (+ 2318us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:56.002483 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:56.002682 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:56.003407 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:56.003425 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:56.003461 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:56.003888 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:56.003902 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.005998 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.006006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:56.008676 (+ 2670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:56.008695 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.011429 (+ 2734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.011436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:56.011829 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:56.011901 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 15:17:56.012106 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 15:17:56.012346 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2040,"thread_start_us":1896,"threads_started":1}
I20260927 15:17:56.013900 28532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:56.000434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37282 (local address 127.10.33.65:33683)
0927 15:17:56.001136 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:56.001146 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:56.002642 (+ 1496us) server_negotiation.cc:407] Connection header received
0927 15:17:56.002892 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:56.002898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:56.003169 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:56.003373 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:56.004102 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.005793 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.008918 (+ 3125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.010575 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.010703 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:56.011200 (+ 497us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:56.013240 (+ 2040us) server_negotiation.cc:299] Negotiation successful
0927 15:17:56.013479 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":329,"threads_started":1}
I20260927 15:17:56.015421 28258 heartbeater.cc:503] Master 127.10.33.126:41611 was elected leader, sending a full tablet report...
I20260927 15:17:56.016283 28535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:56.006553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:35247 (local address 127.0.0.1:51578)
0927 15:17:56.007096 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:56.007112 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:56.007213 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:56.010253 (+ 3040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:56.010259 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:56.010290 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:56.010610 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:56.010620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.012345 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.012351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:56.013931 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:56.013950 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.015524 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.015532 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:56.015719 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:56.015755 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:17:56.015825 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 15:17:56.015965 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":276,"threads_started":1}
I20260927 15:17:56.018560 28536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:56.006627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51578 (local address 127.10.33.67:35247)
0927 15:17:56.009504 (+ 2877us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:56.009514 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:56.009566 (+ 52us) server_negotiation.cc:407] Connection header received
0927 15:17:56.009673 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:56.009682 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:56.010010 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:56.010782 (+ 772us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:56.010983 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.012195 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.014213 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.015295 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.017033 (+ 1738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:56.017511 (+ 478us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:56.017792 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 15:17:56.018184 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2683,"thread_start_us":2536,"threads_started":1}
I20260927 15:17:56.020043 28193 tablet_service.cc:1467] Processing CreateTablet for tablet 9b6203585f7041eb9d6d05d2b052bf04 (DEFAULT_TABLE table=test-table [id=1ee49a4905224df2abddf0d94396f275]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:56.021731 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b6203585f7041eb9d6d05d2b052bf04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:56.023499 28454 tablet_service.cc:1467] Processing CreateTablet for tablet 9b6203585f7041eb9d6d05d2b052bf04 (DEFAULT_TABLE table=test-table [id=1ee49a4905224df2abddf0d94396f275]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:56.025213 28454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b6203585f7041eb9d6d05d2b052bf04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:56.034143 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:56.003053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:36161 (local address 127.0.0.1:48308)
0927 15:17:56.006245 (+ 3192us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:56.006265 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:56.006373 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:56.026504 (+ 20131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:56.026510 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:56.026545 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:56.026899 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:56.026914 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.028917 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.028923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:56.031237 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:56.031260 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.033279 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.033286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:56.033490 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:56.033526 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:17:56.033629 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 15:17:56.033774 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3058,"thread_start_us":2910,"threads_started":1}
I20260927 15:17:56.034449 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:56.003148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48308 (local address 127.10.33.66:36161)
0927 15:17:56.025771 (+ 22623us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:56.025780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:56.025818 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:17:56.025977 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:56.025985 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:56.026258 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:56.026492 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:56.027103 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.028742 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.031532 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.033053 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.033200 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:56.033451 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:56.033709 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 15:17:56.033931 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22448,"thread_start_us":22819,"threads_started":1}
I20260927 15:17:56.036489 28537 tablet_bootstrap.cc:492] T 9b6203585f7041eb9d6d05d2b052bf04 P ac28e997bca34402b439c07b60cb1370: Bootstrap starting.
I20260927 15:17:56.038946 28324 tablet_service.cc:1467] Processing CreateTablet for tablet 9b6203585f7041eb9d6d05d2b052bf04 (DEFAULT_TABLE table=test-table [id=1ee49a4905224df2abddf0d94396f275]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:56.040431 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b6203585f7041eb9d6d05d2b052bf04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:56.041008 28537 tablet_bootstrap.cc:654] T 9b6203585f7041eb9d6d05d2b052bf04 P ac28e997bca34402b439c07b60cb1370: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:56.042484 28537 log.cc:824] T 9b6203585f7041eb9d6d05d2b052bf04 P ac28e997bca34402b439c07b60cb1370: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:56.044938 28538 tablet_bootstrap.cc:492] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6: Bootstrap starting.
I20260927 15:17:56.045925 28537 tablet_bootstrap.cc:492] T 9b6203585f7041eb9d6d05d2b052bf04 P ac28e997bca34402b439c07b60cb1370: No bootstrap required, opened a new log
I20260927 15:17:56.046231 28537 ts_tablet_manager.cc:1397] T 9b6203585f7041eb9d6d05d2b052bf04 P ac28e997bca34402b439c07b60cb1370: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260927 15:17:56.048753 28538 tablet_bootstrap.cc:654] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:56.050113 28538 log.cc:824] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:56.053062 28541 tablet_bootstrap.cc:492] T 9b6203585f7041eb9d6d05d2b052bf04 P 274d568421d0469081781e2b71e30f5f: Bootstrap starting.
I20260927 15:17:56.053722 28538 tablet_bootstrap.cc:492] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6: No bootstrap required, opened a new log
I20260927 15:17:56.054090 28538 ts_tablet_manager.cc:1397] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 15:17:56.056169 28541 tablet_bootstrap.cc:654] T 9b6203585f7041eb9d6d05d2b052bf04 P 274d568421d0469081781e2b71e30f5f: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:56.057153 28541 log.cc:824] T 9b6203585f7041eb9d6d05d2b052bf04 P 274d568421d0469081781e2b71e30f5f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:56.056989 28537 raft_consensus.cc:348] T 9b6203585f7041eb9d6d05d2b052bf04 P ac28e997bca34402b439c07b60cb1370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac28e997bca34402b439c07b60cb1370" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33683 } } peers { permanent_uuid: "274d568421d0469081781e2b71e30f5f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36161 } } peers { permanent_uuid: "b150238ef3524ce48001dfe78dd2abf6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 35247 } }
I20260927 15:17:56.057487 28537 raft_consensus.cc:374] T 9b6203585f7041eb9d6d05d2b052bf04 P ac28e997bca34402b439c07b60cb1370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:56.057621 28537 raft_consensus.cc:729] T 9b6203585f7041eb9d6d05d2b052bf04 P ac28e997bca34402b439c07b60cb1370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac28e997bca34402b439c07b60cb1370, State: Initialized, Role: FOLLOWER
I20260927 15:17:56.058542 28537 consensus_queue.cc:260] T 9b6203585f7041eb9d6d05d2b052bf04 P ac28e997bca34402b439c07b60cb1370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac28e997bca34402b439c07b60cb1370" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33683 } } peers { permanent_uuid: "274d568421d0469081781e2b71e30f5f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36161 } } peers { permanent_uuid: "b150238ef3524ce48001dfe78dd2abf6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 35247 } }
I20260927 15:17:56.060745 28541 tablet_bootstrap.cc:492] T 9b6203585f7041eb9d6d05d2b052bf04 P 274d568421d0469081781e2b71e30f5f: No bootstrap required, opened a new log
I20260927 15:17:56.061075 28541 ts_tablet_manager.cc:1397] T 9b6203585f7041eb9d6d05d2b052bf04 P 274d568421d0469081781e2b71e30f5f: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260927 15:17:56.061244 28537 ts_tablet_manager.cc:1428] T 9b6203585f7041eb9d6d05d2b052bf04 P ac28e997bca34402b439c07b60cb1370: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 15:17:56.064680 28538 raft_consensus.cc:348] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac28e997bca34402b439c07b60cb1370" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33683 } } peers { permanent_uuid: "274d568421d0469081781e2b71e30f5f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36161 } } peers { permanent_uuid: "b150238ef3524ce48001dfe78dd2abf6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 35247 } }
I20260927 15:17:56.065069 28538 raft_consensus.cc:374] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:56.065172 28538 raft_consensus.cc:729] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b150238ef3524ce48001dfe78dd2abf6, State: Initialized, Role: FOLLOWER
I20260927 15:17:56.065950 28538 consensus_queue.cc:260] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac28e997bca34402b439c07b60cb1370" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33683 } } peers { permanent_uuid: "274d568421d0469081781e2b71e30f5f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36161 } } peers { permanent_uuid: "b150238ef3524ce48001dfe78dd2abf6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 35247 } }
I20260927 15:17:56.067088 28519 heartbeater.cc:503] Master 127.10.33.126:41611 was elected leader, sending a full tablet report...
I20260927 15:17:56.067552 28538 ts_tablet_manager.cc:1428] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 15:17:56.070034 28541 raft_consensus.cc:348] T 9b6203585f7041eb9d6d05d2b052bf04 P 274d568421d0469081781e2b71e30f5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac28e997bca34402b439c07b60cb1370" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33683 } } peers { permanent_uuid: "274d568421d0469081781e2b71e30f5f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36161 } } peers { permanent_uuid: "b150238ef3524ce48001dfe78dd2abf6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 35247 } }
I20260927 15:17:56.070530 28541 raft_consensus.cc:374] T 9b6203585f7041eb9d6d05d2b052bf04 P 274d568421d0469081781e2b71e30f5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:56.070618 28541 raft_consensus.cc:729] T 9b6203585f7041eb9d6d05d2b052bf04 P 274d568421d0469081781e2b71e30f5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274d568421d0469081781e2b71e30f5f, State: Initialized, Role: FOLLOWER
I20260927 15:17:56.071260 28541 consensus_queue.cc:260] T 9b6203585f7041eb9d6d05d2b052bf04 P 274d568421d0469081781e2b71e30f5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac28e997bca34402b439c07b60cb1370" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33683 } } peers { permanent_uuid: "274d568421d0469081781e2b71e30f5f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36161 } } peers { permanent_uuid: "b150238ef3524ce48001dfe78dd2abf6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 35247 } }
I20260927 15:17:56.072295 28389 heartbeater.cc:503] Master 127.10.33.126:41611 was elected leader, sending a full tablet report...
I20260927 15:17:56.073019 28541 ts_tablet_manager.cc:1428] T 9b6203585f7041eb9d6d05d2b052bf04 P 274d568421d0469081781e2b71e30f5f: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260927 15:17:56.154409 28544 raft_consensus.cc:482] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:17:56.154740 28544 raft_consensus.cc:504] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac28e997bca34402b439c07b60cb1370" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33683 } } peers { permanent_uuid: "274d568421d0469081781e2b71e30f5f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36161 } } peers { permanent_uuid: "b150238ef3524ce48001dfe78dd2abf6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 35247 } }
I20260927 15:17:56.156373 28544 leader_election.cc:288] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac28e997bca34402b439c07b60cb1370 (127.10.33.65:33683), 274d568421d0469081781e2b71e30f5f (127.10.33.66:36161)
I20260927 15:17:56.162897 28532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:56.156808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:34747 (local address 127.10.33.65:33683)
0927 15:17:56.156998 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:56.157004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:56.157024 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:17:56.157188 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:56.157192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:56.157264 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:56.157354 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:56.158022 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.158728 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.160724 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.161937 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.162060 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:56.162127 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:56.162527 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 15:17:56.162655 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:17:56.163024 28521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:56.156710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33683 (local address 127.10.33.67:34747)
0927 15:17:56.156894 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:56.156916 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:56.157052 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:56.157534 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:56.157539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:56.157565 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:56.157809 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:56.157821 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.158882 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.158887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:56.160567 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:56.160581 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.162107 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.162113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:56.162291 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:56.162359 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 15:17:56.162464 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 15:17:56.162668 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":21}
I20260927 15:17:56.164032 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:56.157250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:42609 (local address 127.10.33.66:36161)
0927 15:17:56.157488 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:56.157494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:56.157800 (+ 306us) server_negotiation.cc:407] Connection header received
0927 15:17:56.157980 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:56.157987 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:56.158064 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:56.158151 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:56.158670 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.159678 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.161809 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.162948 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.163106 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:56.163222 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:56.163706 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 15:17:56.163841 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:17:56.164162 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:56.157087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:36161 (local address 127.10.33.67:42609)
0927 15:17:56.157690 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:56.157707 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:56.157834 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:56.158181 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:56.158185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:56.158209 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:56.158529 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:56.158538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.159837 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.159841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:56.161605 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:56.161620 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.163339 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.163345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:56.163523 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:56.163553 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 15:17:56.163642 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 15:17:56.163768 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":23,"thread_start_us":297,"threads_started":1}
I20260927 15:17:56.164283 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b6203585f7041eb9d6d05d2b052bf04" candidate_uuid: "b150238ef3524ce48001dfe78dd2abf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac28e997bca34402b439c07b60cb1370" is_pre_election: true
I20260927 15:17:56.164908 28213 raft_consensus.cc:2454] T 9b6203585f7041eb9d6d05d2b052bf04 P ac28e997bca34402b439c07b60cb1370 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b150238ef3524ce48001dfe78dd2abf6 in term 0.
I20260927 15:17:56.165138 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b6203585f7041eb9d6d05d2b052bf04" candidate_uuid: "b150238ef3524ce48001dfe78dd2abf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274d568421d0469081781e2b71e30f5f" is_pre_election: true
I20260927 15:17:56.165560 28408 leader_election.cc:302] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac28e997bca34402b439c07b60cb1370, b150238ef3524ce48001dfe78dd2abf6; no voters:
I20260927 15:17:56.165637 28344 raft_consensus.cc:2454] T 9b6203585f7041eb9d6d05d2b052bf04 P 274d568421d0469081781e2b71e30f5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b150238ef3524ce48001dfe78dd2abf6 in term 0.
I20260927 15:17:56.166003 28544 raft_consensus.cc:2781] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:17:56.166169 28544 raft_consensus.cc:482] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:17:56.166275 28544 raft_consensus.cc:3037] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:56.168408 28544 raft_consensus.cc:504] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac28e997bca34402b439c07b60cb1370" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33683 } } peers { permanent_uuid: "274d568421d0469081781e2b71e30f5f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36161 } } peers { permanent_uuid: "b150238ef3524ce48001dfe78dd2abf6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 35247 } }
I20260927 15:17:56.169035 28544 leader_election.cc:288] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6 [CANDIDATE]: Term 1 election: Requested vote from peers ac28e997bca34402b439c07b60cb1370 (127.10.33.65:33683), 274d568421d0469081781e2b71e30f5f (127.10.33.66:36161)
I20260927 15:17:56.169520 28213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b6203585f7041eb9d6d05d2b052bf04" candidate_uuid: "b150238ef3524ce48001dfe78dd2abf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac28e997bca34402b439c07b60cb1370"
I20260927 15:17:56.169556 28344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b6203585f7041eb9d6d05d2b052bf04" candidate_uuid: "b150238ef3524ce48001dfe78dd2abf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274d568421d0469081781e2b71e30f5f"
I20260927 15:17:56.169709 28213 raft_consensus.cc:3037] T 9b6203585f7041eb9d6d05d2b052bf04 P ac28e997bca34402b439c07b60cb1370 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:56.169716 28344 raft_consensus.cc:3037] T 9b6203585f7041eb9d6d05d2b052bf04 P 274d568421d0469081781e2b71e30f5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:56.171893 28344 raft_consensus.cc:2454] T 9b6203585f7041eb9d6d05d2b052bf04 P 274d568421d0469081781e2b71e30f5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b150238ef3524ce48001dfe78dd2abf6 in term 1.
I20260927 15:17:56.171901 28213 raft_consensus.cc:2454] T 9b6203585f7041eb9d6d05d2b052bf04 P ac28e997bca34402b439c07b60cb1370 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b150238ef3524ce48001dfe78dd2abf6 in term 1.
I20260927 15:17:56.172253 28408 leader_election.cc:302] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac28e997bca34402b439c07b60cb1370, b150238ef3524ce48001dfe78dd2abf6; no voters:
I20260927 15:17:56.172531 28544 raft_consensus.cc:2781] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:17:56.172984 28544 raft_consensus.cc:686] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6 [term 1 LEADER]: Becoming Leader. State: Replica: b150238ef3524ce48001dfe78dd2abf6, State: Running, Role: LEADER
I20260927 15:17:56.173525 28544 consensus_queue.cc:237] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac28e997bca34402b439c07b60cb1370" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33683 } } peers { permanent_uuid: "274d568421d0469081781e2b71e30f5f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36161 } } peers { permanent_uuid: "b150238ef3524ce48001dfe78dd2abf6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 35247 } }
I20260927 15:17:56.178557 28071 catalog_manager.cc:5495] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b150238ef3524ce48001dfe78dd2abf6 (127.10.33.67). New cstate: current_term: 1 leader_uuid: "b150238ef3524ce48001dfe78dd2abf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac28e997bca34402b439c07b60cb1370" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "274d568421d0469081781e2b71e30f5f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b150238ef3524ce48001dfe78dd2abf6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 35247 } health_report { overall_health: HEALTHY } } }
W20260927 15:17:56.189738 28520 tablet.cc:2369] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:17:56.214777 28536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:56.208816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51580 (local address 127.10.33.67:35247)
0927 15:17:56.208987 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:56.208992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:56.209057 (+ 65us) server_negotiation.cc:407] Connection header received
0927 15:17:56.209148 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:56.209152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:56.209231 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:56.209359 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:17:56.209816 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.210640 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.212181 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.212636 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.212746 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:56.213397 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:17:56.214007 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:17:56.214193 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:56.214365 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 15:17:56.214445 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:17:56.223933 28344 raft_consensus.cc:1261] T 9b6203585f7041eb9d6d05d2b052bf04 P 274d568421d0469081781e2b71e30f5f [term 1 FOLLOWER]: Refusing update from remote peer b150238ef3524ce48001dfe78dd2abf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:17:56.223932 28213 raft_consensus.cc:1261] T 9b6203585f7041eb9d6d05d2b052bf04 P ac28e997bca34402b439c07b60cb1370 [term 1 FOLLOWER]: Refusing update from remote peer b150238ef3524ce48001dfe78dd2abf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:17:56.225167 28547 consensus_queue.cc:1035] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac28e997bca34402b439c07b60cb1370" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 33683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:17:56.225661 28544 consensus_queue.cc:1035] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "274d568421d0469081781e2b71e30f5f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:17:56.237632 28390 tablet.cc:2369] T 9b6203585f7041eb9d6d05d2b052bf04 P 274d568421d0469081781e2b71e30f5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:17:56.238095 28550 mvcc.cc:205] Tried to move back new op lower bound from 7333979243401764864 to 7333979243211239424. Current Snapshot: MvccSnapshot[applied={T|T < 7333979243401764864}]
W20260927 15:17:56.240485 28259 tablet.cc:2369] T 9b6203585f7041eb9d6d05d2b052bf04 P ac28e997bca34402b439c07b60cb1370: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:17:56.242797 28552 mvcc.cc:205] Tried to move back new op lower bound from 7333979243401764864 to 7333979243211239424. Current Snapshot: MvccSnapshot[applied={T|T < 7333979243401764864}]
I20260927 15:17:56.251405 28071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:17:56.251720 28071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:17:56.255653 28071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de289cb092ba402aaf0434a1b78cd625: Sending DeleteTablet for 3 replicas of tablet 9b6203585f7041eb9d6d05d2b052bf04
I20260927 15:17:56.256803 28193 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b6203585f7041eb9d6d05d2b052bf04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:56 UTC) from {username='slave'} at 127.0.0.1:37282
I20260927 15:17:56.257098 28324 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b6203585f7041eb9d6d05d2b052bf04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:56 UTC) from {username='slave'} at 127.0.0.1:48308
I20260927 15:17:56.257251 28454 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b6203585f7041eb9d6d05d2b052bf04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:56 UTC) from {username='slave'} at 127.0.0.1:51578
I20260927 15:17:56.257982 28559 tablet_replica.cc:330] T 9b6203585f7041eb9d6d05d2b052bf04 P ac28e997bca34402b439c07b60cb1370: stopping tablet replica
I20260927 15:17:56.258231 28560 tablet_replica.cc:330] T 9b6203585f7041eb9d6d05d2b052bf04 P 274d568421d0469081781e2b71e30f5f: stopping tablet replica
I20260927 15:17:56.258281 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 28131
I20260927 15:17:56.258643 28559 raft_consensus.cc:2229] T 9b6203585f7041eb9d6d05d2b052bf04 P ac28e997bca34402b439c07b60cb1370 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:17:56.258671 28560 raft_consensus.cc:2229] T 9b6203585f7041eb9d6d05d2b052bf04 P 274d568421d0469081781e2b71e30f5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:17:56.259222 28559 raft_consensus.cc:2258] T 9b6203585f7041eb9d6d05d2b052bf04 P ac28e997bca34402b439c07b60cb1370 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:56.259225 28560 raft_consensus.cc:2258] T 9b6203585f7041eb9d6d05d2b052bf04 P 274d568421d0469081781e2b71e30f5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:56.261554 28561 tablet_replica.cc:330] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6: stopping tablet replica
I20260927 15:17:56.262072 28561 raft_consensus.cc:2229] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:17:56.262811 28561 raft_consensus.cc:2258] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:56.263731 28560 ts_tablet_manager.cc:1905] T 9b6203585f7041eb9d6d05d2b052bf04 P 274d568421d0469081781e2b71e30f5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:56.262867 28559 ts_tablet_manager.cc:1905] T 9b6203585f7041eb9d6d05d2b052bf04 P ac28e997bca34402b439c07b60cb1370: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:56.265111 28561 ts_tablet_manager.cc:1905] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:56.272610 28532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:56.259643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37292 (local address 127.10.33.65:33683)
0927 15:17:56.259827 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:56.259833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:56.263830 (+ 3997us) server_negotiation.cc:407] Connection header received
0927 15:17:56.264116 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:56.264122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:56.264235 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:56.264317 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:56.265212 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.266586 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.268531 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.269240 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.269344 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:56.270893 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:56.270904 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:56.270923 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:17:56.271690 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:56.271755 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:17:56.271783 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:56.271889 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:56.272120 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 15:17:56.272243 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:17:56.273831 28253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:17:56.275956 28561 ts_tablet_manager.cc:1918] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:56.276218 28561 log.cc:1196] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/9b6203585f7041eb9d6d05d2b052bf04
I20260927 15:17:56.276896 28561 ts_tablet_manager.cc:1939] T 9b6203585f7041eb9d6d05d2b052bf04 P b150238ef3524ce48001dfe78dd2abf6: Deleting consensus metadata
I20260927 15:17:56.276907 28560 ts_tablet_manager.cc:1918] T 9b6203585f7041eb9d6d05d2b052bf04 P 274d568421d0469081781e2b71e30f5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:56.277145 28560 log.cc:1196] T 9b6203585f7041eb9d6d05d2b052bf04 P 274d568421d0469081781e2b71e30f5f: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/9b6203585f7041eb9d6d05d2b052bf04
I20260927 15:17:56.277741 28560 ts_tablet_manager.cc:1939] T 9b6203585f7041eb9d6d05d2b052bf04 P 274d568421d0469081781e2b71e30f5f: Deleting consensus metadata
I20260927 15:17:56.279296 28057 catalog_manager.cc:4841] TS b150238ef3524ce48001dfe78dd2abf6 (127.10.33.67:35247): tablet 9b6203585f7041eb9d6d05d2b052bf04 (table test-table [id=1ee49a4905224df2abddf0d94396f275]) successfully deleted
I20260927 15:17:56.280937 28057 catalog_manager.cc:4841] TS 274d568421d0469081781e2b71e30f5f (127.10.33.66:36161): tablet 9b6203585f7041eb9d6d05d2b052bf04 (table test-table [id=1ee49a4905224df2abddf0d94396f275]) successfully deleted
I20260927 15:17:56.402608 28559 ts_tablet_manager.cc:1918] T 9b6203585f7041eb9d6d05d2b052bf04 P ac28e997bca34402b439c07b60cb1370: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:56.402920 28559 log.cc:1196] T 9b6203585f7041eb9d6d05d2b052bf04 P ac28e997bca34402b439c07b60cb1370: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/9b6203585f7041eb9d6d05d2b052bf04
I20260927 15:17:56.403401 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 28131
W20260927 15:17:56.417631 28057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.33.65:33683 (error 108)
I20260927 15:17:56.419215 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 28262
I20260927 15:17:56.419734 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:56.419214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33683 (local address 127.0.0.1:37302)
0927 15:17:56.419427 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:56.419568 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 15:17:56.420132 28057 catalog_manager.cc:4584] TS ac28e997bca34402b439c07b60cb1370 (127.10.33.65:33683): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9b6203585f7041eb9d6d05d2b052bf04: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33683: connect: Connection refused (error 111)
I20260927 15:17:56.428017 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:56.420008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48316 (local address 127.10.33.66:36161)
0927 15:17:56.420190 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:56.420196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:56.420216 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:17:56.420371 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:56.420377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:56.420478 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:56.420559 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:56.421178 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.422461 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.424328 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.424993 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.425130 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:56.426381 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:56.426393 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:56.426414 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:17:56.427190 (+ 776us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:56.427255 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:17:56.427282 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:56.427403 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:56.427545 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 15:17:56.427670 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 15:17:56.428809 28384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:17:56.473688 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:56.473226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33683 (local address 127.0.0.1:37314)
0927 15:17:56.473437 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:56.473525 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 15:17:56.474813 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:56.474451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33683 (local address 127.0.0.1:37326)
0927 15:17:56.474605 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:56.474679 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 15:17:56.535640 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:56.535131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33683 (local address 127.0.0.1:37328)
0927 15:17:56.535351 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:56.535451 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 15:17:56.536703 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:56.536369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33683 (local address 127.0.0.1:37342)
0927 15:17:56.536514 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:56.536587 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 15:17:56.576700 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 28262
I20260927 15:17:56.590987 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 28392
I20260927 15:17:56.598659 28536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:56.591793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43110 (local address 127.10.33.67:35247)
0927 15:17:56.591961 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:56.591965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:56.591982 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:17:56.592075 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:56.592079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:56.592157 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:56.592235 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:56.592682 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.593504 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.595342 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:56.595825 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:56.595944 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:56.597106 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:56.597117 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:56.597131 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:17:56.597700 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:56.597766 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:17:56.597787 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:56.597908 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:56.598128 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 15:17:56.598240 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:17:56.599424 28514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:17:56.617543 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:56.616910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33683 (local address 127.0.0.1:35374)
0927 15:17:56.617220 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:56.617342 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260927 15:17:56.619025 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:56.618508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33683 (local address 127.0.0.1:35380)
0927 15:17:56.618772 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:56.618879 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260927 15:17:56.724936 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 28392
I20260927 15:17:56.740494 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 28041
I20260927 15:17:56.741641 28102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:17:56.848839 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:56.848232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:33683 (local address 127.0.0.1:35386)
0927 15:17:56.848474 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:56.848597 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 15:17:56.849252 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 28041
2026-09-27T15:17:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2938 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:17:56 dist-test-slave-f5h6 krb5kdc[28570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:17:56 dist-test-slave-f5h6 krb5kdc[28570](info): set up 2 sockets
Sep 27 15:17:56 dist-test-slave-f5h6 krb5kdc[28570](info): commencing operation
krb5kdc: starting...
W20260927 15:17:57.889931 10373 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:17:58 dist-test-slave-f5h6 krb5kdc[28570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:17:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:17:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:17:58.147516 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:40025
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:40025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:58.474146 28583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:58.474452 28583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:58.474555 28583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:58.482092 28583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:17:58.482216 28583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:58.482275 28583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:58.482337 28583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:17:58.482389 28583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:17:58.492831 28583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:40025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:40025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:17:58.494256 28583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:58.495584 28583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:58.507966 28588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:58.508008 28591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:58.507941 28589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:58.508587 28583 server_base.cc:1034] running on GCE node
I20260927 15:17:58.509465 28583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:58.511101 28583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:58.512324 28583 hybrid_clock.cc:648] HybridClock initialized: now 1790522278512263 us; error 54 us; skew 500 ppm
I20260927 15:17:58.512679 28583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:17:58 dist-test-slave-f5h6 krb5kdc[28570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522278, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:58.516049 28583 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:17:58.517644 28583 webserver.cc:460] Webserver started at http://127.10.33.126:39993/ using document root <none> and password file <none>
I20260927 15:17:58.518424 28583 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:58.518537 28583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:58.518888 28583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:58.521975 28583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "f6a1b80eaa9743818a6a832c6cf97c54"
format_stamp: "Formatted at 2026-09-27 15:17:58 on dist-test-slave-f5h6"
server_key: "21ac4b77cf52cacb62a75492663e2cb6"
server_key_iv: "9f07250ac44d0aa6ac4c874f398fb93f"
server_key_version: "encryptionkey@0"
I20260927 15:17:58.522737 28583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "f6a1b80eaa9743818a6a832c6cf97c54"
format_stamp: "Formatted at 2026-09-27 15:17:58 on dist-test-slave-f5h6"
server_key: "21ac4b77cf52cacb62a75492663e2cb6"
server_key_iv: "9f07250ac44d0aa6ac4c874f398fb93f"
server_key_version: "encryptionkey@0"
I20260927 15:17:58.528043 28583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 15:17:58.531701 28598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:58.533197 28583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:17:58.533390 28583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "f6a1b80eaa9743818a6a832c6cf97c54"
format_stamp: "Formatted at 2026-09-27 15:17:58 on dist-test-slave-f5h6"
server_key: "21ac4b77cf52cacb62a75492663e2cb6"
server_key_iv: "9f07250ac44d0aa6ac4c874f398fb93f"
server_key_version: "encryptionkey@0"
I20260927 15:17:58.533623 28583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:58.552507 28583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:58.557312 28583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:58.557632 28583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:58.577159 28583 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:40025
I20260927 15:17:58.577188 28649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:40025 every 8 connection(s)
I20260927 15:17:58.578665 28583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:17:58.581472 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 28583
I20260927 15:17:58.581624 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:17:58.582211 10373 external_mini_cluster.cc:1428] Setting key 0b86615de578e0e1488d7eb84c14069c
I20260927 15:17:58.582723 28650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:58.597313 28650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54: Bootstrap starting.
Sep 27 15:17:58 dist-test-slave-f5h6 krb5kdc[28570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:17:58.601198 28650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:58.602581 28650 log.cc:824] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:58.606058 28650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54: No bootstrap required, opened a new log
I20260927 15:17:58.608182 28653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:58.584048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44744 (local address 127.10.33.126:40025)
0927 15:17:58.584824 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:58.584836 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:58.584875 (+ 39us) server_negotiation.cc:407] Connection header received
0927 15:17:58.587295 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:58.587319 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:58.587829 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:58.588202 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:58.590240 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:58.591562 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:58.593369 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:58.594231 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:58.594375 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:58.599354 (+ 4979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:58.599371 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:58.599393 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:58.599456 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:58.603432 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:58.604281 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:58.604290 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:58.604298 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:58.604533 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:58.604909 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:58.604914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:58.604917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:58.605653 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:58.605956 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:58.606385 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 15:17:58.606863 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":349,"threads_started":1}
I20260927 15:17:58.615087 28650 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a1b80eaa9743818a6a832c6cf97c54" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40025 } }
I20260927 15:17:58.615471 28650 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:58.615548 28650 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a1b80eaa9743818a6a832c6cf97c54, State: Initialized, Role: FOLLOWER
I20260927 15:17:58.616084 28650 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a1b80eaa9743818a6a832c6cf97c54" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40025 } }
I20260927 15:17:58.616292 28650 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:17:58.616389 28650 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:17:58.616519 28650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:58.618799 28650 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a1b80eaa9743818a6a832c6cf97c54" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40025 } }
I20260927 15:17:58.619298 28650 leader_election.cc:302] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6a1b80eaa9743818a6a832c6cf97c54; no voters:
I20260927 15:17:58.619871 28650 leader_election.cc:288] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:17:58.619993 28655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:17:58.620605 28655 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54 [term 1 LEADER]: Becoming Leader. State: Replica: f6a1b80eaa9743818a6a832c6cf97c54, State: Running, Role: LEADER
I20260927 15:17:58.621281 28655 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a1b80eaa9743818a6a832c6cf97c54" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40025 } }
I20260927 15:17:58.622457 28650 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:17:58.623927 28657 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6a1b80eaa9743818a6a832c6cf97c54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a1b80eaa9743818a6a832c6cf97c54" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40025 } } }
I20260927 15:17:58.623901 28656 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6a1b80eaa9743818a6a832c6cf97c54. Latest consensus state: current_term: 1 leader_uuid: "f6a1b80eaa9743818a6a832c6cf97c54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a1b80eaa9743818a6a832c6cf97c54" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 40025 } } }
I20260927 15:17:58.624341 28657 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54 [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:58.624341 28656 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54 [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:58.628580 28660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:17:58.634775 28660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:17:58.645042 28660 catalog_manager.cc:1347] Generated new cluster ID: e2165fe8308e4907bafa7f8622a6377e
I20260927 15:17:58.645171 28660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:17:58.675220 28660 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:17:58.677126 28660 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:17:58.693166 28660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54: Generated new TSK 0
I20260927 15:17:58.694073 28660 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.33.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
I20260927 15:17:58.765273 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:40025
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:59.116284 28676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:59.116586 28676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:59.116685 28676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:59.124408 28676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:59.124531 28676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:59.124652 28676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:17:59.134963 28676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:40025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:17:59.136247 28676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:59.137562 28676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:59.150846 28682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:59.150913 28681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:59.150879 28684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:59.151747 28676 server_base.cc:1034] running on GCE node
I20260927 15:17:59.152329 28676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:59.153337 28676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:59.154553 28676 hybrid_clock.cc:648] HybridClock initialized: now 1790522279154482 us; error 48 us; skew 500 ppm
I20260927 15:17:59.154872 28676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:17:59 dist-test-slave-f5h6 krb5kdc[28570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522279, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:59.158222 28676 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:17:59.159834 28676 webserver.cc:460] Webserver started at http://127.10.33.65:44985/ using document root <none> and password file <none>
I20260927 15:17:59.160534 28676 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:59.160660 28676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:59.160965 28676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:59.164053 28676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "a714a4227be54aa29ca2981f2bfe4a6c"
format_stamp: "Formatted at 2026-09-27 15:17:59 on dist-test-slave-f5h6"
server_key: "0450c236c105ea86605798ad04e7b834"
server_key_iv: "64d1cb61a0c7d6c9e153a379afa92c53"
server_key_version: "encryptionkey@0"
I20260927 15:17:59.164837 28676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "a714a4227be54aa29ca2981f2bfe4a6c"
format_stamp: "Formatted at 2026-09-27 15:17:59 on dist-test-slave-f5h6"
server_key: "0450c236c105ea86605798ad04e7b834"
server_key_iv: "64d1cb61a0c7d6c9e153a379afa92c53"
server_key_version: "encryptionkey@0"
I20260927 15:17:59.170555 28676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 15:17:59.174499 28691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:59.176083 28676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:17:59.176262 28676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "a714a4227be54aa29ca2981f2bfe4a6c"
format_stamp: "Formatted at 2026-09-27 15:17:59 on dist-test-slave-f5h6"
server_key: "0450c236c105ea86605798ad04e7b834"
server_key_iv: "64d1cb61a0c7d6c9e153a379afa92c53"
server_key_version: "encryptionkey@0"
I20260927 15:17:59.176476 28676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:59.198627 28676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:59.202785 28676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:59.203073 28676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:59.204092 28676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:59.205386 28676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:59.205471 28676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:59.205601 28676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:59.205654 28676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:59.237565 28676 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:46725
I20260927 15:17:59.237601 28803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:46725 every 8 connection(s)
I20260927 15:17:59.238938 28676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:17:59.244572 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 28676
I20260927 15:17:59.244757 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:17:59.245255 10373 external_mini_cluster.cc:1428] Setting key 2e7ae81ceb2fc0ac4a7db2872ecd921e
Sep 27 15:17:59 dist-test-slave-f5h6 krb5kdc[28570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522279, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:17:59.260804 28806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:59.241587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:35989 (local address 127.10.33.126:40025)
0927 15:17:59.242036 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:59.242041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:59.243005 (+ 964us) server_negotiation.cc:407] Connection header received
0927 15:17:59.245125 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:59.245130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:59.245216 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:59.245321 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:59.247136 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:59.247912 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:59.250140 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:59.250570 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:59.250676 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:59.254738 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:59.254743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:59.254746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:59.254783 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:59.257365 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:59.258298 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:59.258303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:59.258305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:59.258450 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:59.258967 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:59.258972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:59.258974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:59.259299 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:59.259443 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:59.260291 (+ 848us) server_negotiation.cc:299] Negotiation successful
0927 15:17:59.260501 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":219,"threads_started":1}
I20260927 15:17:59.262110 28807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:59.242112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40025 (local address 127.10.33.65:35989)
0927 15:17:59.242837 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:59.242877 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:59.244756 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:59.245604 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:59.245616 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:59.246350 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:59.246940 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:59.246963 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:59.248087 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:59.248093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:59.249966 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:59.249981 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:59.250792 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:59.250798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:59.251066 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:59.251757 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:59.251790 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:59.254472 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:59.257512 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:59.257521 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:59.257537 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:59.258129 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:59.258558 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:59.258563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:59.258566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:59.258787 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:59.259427 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:59.259434 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:59.259850 (+ 416us) client_negotiation.cc:769] Sending connection context
0927 15:17:59.260137 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 15:17:59.260684 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":316,"threads_started":1}
I20260927 15:17:59.263880 28804 heartbeater.cc:347] Connected to a master server at 127.10.33.126:40025
I20260927 15:17:59.264196 28804 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:59.265033 28804 heartbeater.cc:511] Master 127.10.33.126:40025 requested a full tablet report, sending...
I20260927 15:17:59.267354 28615 ts_manager.cc:194] Registered new tserver with Master: a714a4227be54aa29ca2981f2bfe4a6c (127.10.33.65:46725)
I20260927 15:17:59.268813 28615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:35989
WARNING: no policy specified for kudu/127.10.33.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
I20260927 15:17:59.301919 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:40025
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:59.633971 28810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:59.634265 28810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:59.634373 28810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:59.641638 28810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:59.641757 28810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:59.641908 28810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:17:59.652653 28810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:40025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:17:59.654008 28810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:59.655366 28810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:59.669015 28815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:59.668968 28816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:59.668946 28818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:59.669770 28810 server_base.cc:1034] running on GCE node
I20260927 15:17:59.670408 28810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:59.671402 28810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:59.672783 28810 hybrid_clock.cc:648] HybridClock initialized: now 1790522279672724 us; error 47 us; skew 500 ppm
I20260927 15:17:59.673116 28810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:17:59 dist-test-slave-f5h6 krb5kdc[28570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522279, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:59.682045 28810 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:17:59.683705 28810 webserver.cc:460] Webserver started at http://127.10.33.66:45021/ using document root <none> and password file <none>
I20260927 15:17:59.684422 28810 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:59.684530 28810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:59.685017 28810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:59.688191 28810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "f4d5072d773d4bb59347985fb601fe31"
format_stamp: "Formatted at 2026-09-27 15:17:59 on dist-test-slave-f5h6"
server_key: "408ab1587e5a18a92a36c1bebf1ee956"
server_key_iv: "a394d89bee86b9bb491848f58bab4439"
server_key_version: "encryptionkey@0"
I20260927 15:17:59.689025 28810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "f4d5072d773d4bb59347985fb601fe31"
format_stamp: "Formatted at 2026-09-27 15:17:59 on dist-test-slave-f5h6"
server_key: "408ab1587e5a18a92a36c1bebf1ee956"
server_key_iv: "a394d89bee86b9bb491848f58bab4439"
server_key_version: "encryptionkey@0"
I20260927 15:17:59.694670 28810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 15:17:59.698493 28825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:59.699942 28810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:17:59.700103 28810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "f4d5072d773d4bb59347985fb601fe31"
format_stamp: "Formatted at 2026-09-27 15:17:59 on dist-test-slave-f5h6"
server_key: "408ab1587e5a18a92a36c1bebf1ee956"
server_key_iv: "a394d89bee86b9bb491848f58bab4439"
server_key_version: "encryptionkey@0"
I20260927 15:17:59.700304 28810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:59.733604 28810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:59.738420 28810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:59.738716 28810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:59.739706 28810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:59.741139 28810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:59.741210 28810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:59.741297 28810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:59.741348 28810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:59.771425 28810 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:35095
I20260927 15:17:59.771474 28937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:35095 every 8 connection(s)
I20260927 15:17:59.772766 28810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:17:59.777504 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 28810
I20260927 15:17:59.777704 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:17:59.778287 10373 external_mini_cluster.cc:1428] Setting key 6aa09b7254703283001ceb949534c37c
Sep 27 15:17:59 dist-test-slave-f5h6 krb5kdc[28570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522279, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:17:59.795887 28940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:59.775506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:38653 (local address 127.10.33.126:40025)
0927 15:17:59.775931 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:59.775935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:59.777019 (+ 1084us) server_negotiation.cc:407] Connection header received
0927 15:17:59.779403 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:59.779409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:59.779495 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:59.779616 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:59.781768 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:59.782744 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:59.784980 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:59.785566 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:59.785686 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:59.789963 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:59.789968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:59.789971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:59.790010 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:59.792526 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:59.793332 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:59.793336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:59.793338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:59.793475 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:59.793950 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:59.793955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:59.793958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:59.794264 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:59.794402 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:59.795268 (+ 866us) server_negotiation.cc:299] Negotiation successful
0927 15:17:59.795510 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":203,"threads_started":1}
I20260927 15:17:59.796862 28941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:59.776105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40025 (local address 127.10.33.66:38653)
0927 15:17:59.776840 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:59.776879 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:59.779135 (+ 2256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:59.779929 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:59.779941 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:59.780697 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:59.781547 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:59.781575 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:59.782947 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:59.782953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:59.784783 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:59.784800 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:59.785812 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:59.785819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:59.786158 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:59.786939 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:59.786969 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:59.789667 (+ 2698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:59.792702 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:59.792713 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:59.792729 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:59.793199 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:59.793598 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:59.793603 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:59.793606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:59.793814 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:59.794393 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:59.794401 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:59.794778 (+ 377us) client_negotiation.cc:769] Sending connection context
0927 15:17:59.795076 (+ 298us) client_negotiation.cc:240] Negotiation successful
0927 15:17:59.795651 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":289,"threads_started":1}
I20260927 15:17:59.798422 28938 heartbeater.cc:347] Connected to a master server at 127.10.33.126:40025
I20260927 15:17:59.798749 28938 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:59.799538 28938 heartbeater.cc:511] Master 127.10.33.126:40025 requested a full tablet report, sending...
I20260927 15:17:59.801121 28615 ts_manager.cc:194] Registered new tserver with Master: f4d5072d773d4bb59347985fb601fe31 (127.10.33.66:35095)
I20260927 15:17:59.802001 28615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:38653
WARNING: no policy specified for kudu/127.10.33.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
I20260927 15:17:59.839280 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:40025
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:00.171825 28944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:00.172115 28944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:00.172211 28944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:00.179416 28944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:00.179558 28944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:00.179702 28944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:18:00.189716 28944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:40025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:00.190982 28944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:00.192253 28944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:00.205469 28950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:00.205539 28952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:00.205505 28949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:00.206257 28944 server_base.cc:1034] running on GCE node
I20260927 15:18:00.206883 28944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:00.207849 28944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:00.209105 28944 hybrid_clock.cc:648] HybridClock initialized: now 1790522280209041 us; error 56 us; skew 500 ppm
I20260927 15:18:00.209430 28944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:00 dist-test-slave-f5h6 krb5kdc[28570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522280, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:00.212695 28944 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:18:00.214287 28944 webserver.cc:460] Webserver started at http://127.10.33.67:45159/ using document root <none> and password file <none>
I20260927 15:18:00.214975 28944 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:00.215097 28944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:00.215409 28944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:00.218444 28944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "ea86079091d449219270441e7075cd51"
format_stamp: "Formatted at 2026-09-27 15:18:00 on dist-test-slave-f5h6"
server_key: "165a2c26654c9b1f00f9246538bcc283"
server_key_iv: "9871a6a6126232303b28d3083de5107d"
server_key_version: "encryptionkey@0"
I20260927 15:18:00.219209 28944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "ea86079091d449219270441e7075cd51"
format_stamp: "Formatted at 2026-09-27 15:18:00 on dist-test-slave-f5h6"
server_key: "165a2c26654c9b1f00f9246538bcc283"
server_key_iv: "9871a6a6126232303b28d3083de5107d"
server_key_version: "encryptionkey@0"
I20260927 15:18:00.224553 28944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:18:00.228102 28959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:00.229553 28944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:18:00.229722 28944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "ea86079091d449219270441e7075cd51"
format_stamp: "Formatted at 2026-09-27 15:18:00 on dist-test-slave-f5h6"
server_key: "165a2c26654c9b1f00f9246538bcc283"
server_key_iv: "9871a6a6126232303b28d3083de5107d"
server_key_version: "encryptionkey@0"
I20260927 15:18:00.229971 28944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:00.244379 28944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:00.248694 28944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:00.248973 28944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:00.249966 28944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:18:00.251313 28944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:18:00.251392 28944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:00.251510 28944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:18:00.251564 28944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:00.272209 28804 heartbeater.cc:503] Master 127.10.33.126:40025 was elected leader, sending a full tablet report...
I20260927 15:18:00.282757 28944 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:34395
I20260927 15:18:00.282812 29071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:34395 every 8 connection(s)
I20260927 15:18:00.284070 28944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:18:00.294215 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 28944
I20260927 15:18:00.294382 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:18:00.294955 10373 external_mini_cluster.cc:1428] Setting key 3c70060c4f66b1352ad30e4f1296e8a9
Sep 27 15:18:00 dist-test-slave-f5h6 krb5kdc[28570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522280, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:18:00.307543 28940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.286728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:44559 (local address 127.10.33.126:40025)
0927 15:18:00.286929 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:00.286934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:00.288211 (+ 1277us) server_negotiation.cc:407] Connection header received
0927 15:18:00.290418 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:00.290428 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:00.290570 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:00.290676 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:00.292696 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.293444 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.295672 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.296422 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.296582 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:00.301174 (+ 4592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:00.301180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:00.301183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:00.301234 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:00.303908 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:00.304836 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:00.304840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:00.304843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:00.304982 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:00.305593 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:00.305597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:00.305600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:00.306082 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:00.306209 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:00.307082 (+ 873us) server_negotiation.cc:299] Negotiation successful
0927 15:18:00.307301 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:18:00.308480 29074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.287218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:40025 (local address 127.10.33.67:44559)
0927 15:18:00.288017 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:00.288067 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:00.290013 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:00.291003 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:00.291014 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:00.291804 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:00.292458 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:00.292479 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.293663 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.293668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:00.295214 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:00.295232 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.296755 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.296763 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:00.297090 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:00.297957 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:00.297982 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:00.300906 (+ 2924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:00.304061 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:00.304076 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:00.304091 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:00.304655 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:00.305203 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:00.305207 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:00.305210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:00.305424 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:00.306231 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:00.306238 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:00.306623 (+ 385us) client_negotiation.cc:769] Sending connection context
0927 15:18:00.306883 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 15:18:00.307322 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":319,"threads_started":1}
I20260927 15:18:00.310271 29072 heartbeater.cc:347] Connected to a master server at 127.10.33.126:40025
I20260927 15:18:00.310613 29072 heartbeater.cc:464] Registering TS with master...
I20260927 15:18:00.311510 29072 heartbeater.cc:511] Master 127.10.33.126:40025 requested a full tablet report, sending...
I20260927 15:18:00.312908 28615 ts_manager.cc:194] Registered new tserver with Master: ea86079091d449219270441e7075cd51 (127.10.33.67:34395)
I20260927 15:18:00.313704 28615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:44559
I20260927 15:18:00.323818 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:18:00.342281 28940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.327929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44758 (local address 127.10.33.126:40025)
0927 15:18:00.328187 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:00.328195 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:00.328471 (+ 276us) server_negotiation.cc:407] Connection header received
0927 15:18:00.328671 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:00.328677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:00.328745 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:00.328830 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:00.330268 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.331007 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.332516 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.332955 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.333074 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:00.336098 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:00.336104 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:00.336107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:00.336150 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:00.339079 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:00.340037 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:00.340042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:00.340046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:00.340192 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:00.340648 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:00.340652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:00.340654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:00.341000 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:00.341268 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:00.341694 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 15:18:00.341885 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:18:00.346460 28615 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:18:00.353544 28615 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:18:00.376130 29085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.366189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56902 (local address 127.10.33.66:35095)
0927 15:18:00.367684 (+ 1495us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:00.367693 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:00.367720 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:18:00.367822 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:00.367829 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:00.368084 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:00.368306 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:00.369266 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.370823 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.373477 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.374820 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.374955 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:00.375182 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:00.375627 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 15:18:00.375799 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1341,"thread_start_us":1197,"threads_started":1}
I20260927 15:18:00.376209 29086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.367313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:34395 (local address 127.0.0.1:36598)
0927 15:18:00.367976 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:00.367994 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:00.368101 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:00.369255 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:00.369260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:00.369288 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:00.369638 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:00.369651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.371692 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.371696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:00.373138 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:00.373155 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.375106 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.375113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:00.375312 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:00.375404 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 15:18:00.375497 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 15:18:00.375717 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":1,"thread_start_us":426,"threads_started":1}
I20260927 15:18:00.376257 29084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.366064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:35095 (local address 127.0.0.1:56902)
0927 15:18:00.366812 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:00.366862 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:00.367023 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:00.368578 (+ 1555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:00.368586 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:00.368639 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:00.369090 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:00.369101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.370960 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.370965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:00.373269 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:00.373285 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.375034 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.375040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:00.375391 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:00.375424 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:18:00.375547 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 15:18:00.375732 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":133,"thread_start_us":331,"threads_started":1}
I20260927 15:18:00.377628 29087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.367785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36598 (local address 127.10.33.67:34395)
0927 15:18:00.368649 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:00.368657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:00.368690 (+ 33us) server_negotiation.cc:407] Connection header received
0927 15:18:00.368792 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:00.368800 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:00.369028 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:00.369834 (+ 806us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:00.370132 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.371569 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.373363 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.374858 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.376783 (+ 1925us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:00.377035 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:00.377114 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 15:18:00.377277 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":522,"threads_started":1}
I20260927 15:18:00.377961 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.368046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:46725 (local address 127.0.0.1:57586)
0927 15:18:00.368965 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:00.368986 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:00.369101 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:00.370772 (+ 1671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:00.370777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:00.370810 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:00.371211 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:00.371226 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.373093 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.373099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:00.375248 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:00.375262 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.377035 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.377042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:00.377213 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:00.377246 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:18:00.377338 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 15:18:00.377488 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":7,"thread_start_us":746,"threads_started":1}
I20260927 15:18:00.378644 29089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.368602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.10.33.65:46725)
0927 15:18:00.370057 (+ 1455us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:00.370065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:00.370091 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:18:00.370199 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:00.370205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:00.370420 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:00.370588 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:00.371396 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.372935 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.375424 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.376814 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.376963 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:00.377182 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:00.377389 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 15:18:00.377538 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1303,"thread_start_us":752,"threads_started":1}
I20260927 15:18:00.380831 28873 tablet_service.cc:1467] Processing CreateTablet for tablet 82d9fd20b6c6490dacf9fc06bc8be060 (DEFAULT_TABLE table=test-table [id=7b4c69f6864a45eca33a6fb0fb3c1d2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:18:00.381032 29007 tablet_service.cc:1467] Processing CreateTablet for tablet 82d9fd20b6c6490dacf9fc06bc8be060 (DEFAULT_TABLE table=test-table [id=7b4c69f6864a45eca33a6fb0fb3c1d2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:18:00.382340 29007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82d9fd20b6c6490dacf9fc06bc8be060. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:00.382416 28873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82d9fd20b6c6490dacf9fc06bc8be060. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:00.382417 28739 tablet_service.cc:1467] Processing CreateTablet for tablet 82d9fd20b6c6490dacf9fc06bc8be060 (DEFAULT_TABLE table=test-table [id=7b4c69f6864a45eca33a6fb0fb3c1d2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:18:00.383937 28739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82d9fd20b6c6490dacf9fc06bc8be060. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:00.394402 29090 tablet_bootstrap.cc:492] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51: Bootstrap starting.
I20260927 15:18:00.396653 29091 tablet_bootstrap.cc:492] T 82d9fd20b6c6490dacf9fc06bc8be060 P f4d5072d773d4bb59347985fb601fe31: Bootstrap starting.
I20260927 15:18:00.398114 29090 tablet_bootstrap.cc:654] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:00.399531 29092 tablet_bootstrap.cc:492] T 82d9fd20b6c6490dacf9fc06bc8be060 P a714a4227be54aa29ca2981f2bfe4a6c: Bootstrap starting.
I20260927 15:18:00.400307 29091 tablet_bootstrap.cc:654] T 82d9fd20b6c6490dacf9fc06bc8be060 P f4d5072d773d4bb59347985fb601fe31: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:00.401022 29090 log.cc:824] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:00.401685 29091 log.cc:824] T 82d9fd20b6c6490dacf9fc06bc8be060 P f4d5072d773d4bb59347985fb601fe31: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:00.402897 29092 tablet_bootstrap.cc:654] T 82d9fd20b6c6490dacf9fc06bc8be060 P a714a4227be54aa29ca2981f2bfe4a6c: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:00.403903 29092 log.cc:824] T 82d9fd20b6c6490dacf9fc06bc8be060 P a714a4227be54aa29ca2981f2bfe4a6c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:00.404533 29091 tablet_bootstrap.cc:492] T 82d9fd20b6c6490dacf9fc06bc8be060 P f4d5072d773d4bb59347985fb601fe31: No bootstrap required, opened a new log
I20260927 15:18:00.404541 29090 tablet_bootstrap.cc:492] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51: No bootstrap required, opened a new log
I20260927 15:18:00.404824 29090 ts_tablet_manager.cc:1397] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260927 15:18:00.404824 29091 ts_tablet_manager.cc:1397] T 82d9fd20b6c6490dacf9fc06bc8be060 P f4d5072d773d4bb59347985fb601fe31: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 15:18:00.406486 29092 tablet_bootstrap.cc:492] T 82d9fd20b6c6490dacf9fc06bc8be060 P a714a4227be54aa29ca2981f2bfe4a6c: No bootstrap required, opened a new log
I20260927 15:18:00.406739 29092 ts_tablet_manager.cc:1397] T 82d9fd20b6c6490dacf9fc06bc8be060 P a714a4227be54aa29ca2981f2bfe4a6c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260927 15:18:00.414783 29092 raft_consensus.cc:348] T 82d9fd20b6c6490dacf9fc06bc8be060 P a714a4227be54aa29ca2981f2bfe4a6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d5072d773d4bb59347985fb601fe31" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35095 } } peers { permanent_uuid: "ea86079091d449219270441e7075cd51" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34395 } } peers { permanent_uuid: "a714a4227be54aa29ca2981f2bfe4a6c" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 46725 } }
I20260927 15:18:00.415212 29092 raft_consensus.cc:374] T 82d9fd20b6c6490dacf9fc06bc8be060 P a714a4227be54aa29ca2981f2bfe4a6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:00.415328 29092 raft_consensus.cc:729] T 82d9fd20b6c6490dacf9fc06bc8be060 P a714a4227be54aa29ca2981f2bfe4a6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a714a4227be54aa29ca2981f2bfe4a6c, State: Initialized, Role: FOLLOWER
I20260927 15:18:00.415970 29092 consensus_queue.cc:260] T 82d9fd20b6c6490dacf9fc06bc8be060 P a714a4227be54aa29ca2981f2bfe4a6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d5072d773d4bb59347985fb601fe31" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35095 } } peers { permanent_uuid: "ea86079091d449219270441e7075cd51" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34395 } } peers { permanent_uuid: "a714a4227be54aa29ca2981f2bfe4a6c" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 46725 } }
I20260927 15:18:00.415906 29091 raft_consensus.cc:348] T 82d9fd20b6c6490dacf9fc06bc8be060 P f4d5072d773d4bb59347985fb601fe31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d5072d773d4bb59347985fb601fe31" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35095 } } peers { permanent_uuid: "ea86079091d449219270441e7075cd51" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34395 } } peers { permanent_uuid: "a714a4227be54aa29ca2981f2bfe4a6c" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 46725 } }
I20260927 15:18:00.415906 29090 raft_consensus.cc:348] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d5072d773d4bb59347985fb601fe31" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35095 } } peers { permanent_uuid: "ea86079091d449219270441e7075cd51" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34395 } } peers { permanent_uuid: "a714a4227be54aa29ca2981f2bfe4a6c" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 46725 } }
I20260927 15:18:00.416371 29090 raft_consensus.cc:374] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:00.416397 29091 raft_consensus.cc:374] T 82d9fd20b6c6490dacf9fc06bc8be060 P f4d5072d773d4bb59347985fb601fe31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:00.416493 29091 raft_consensus.cc:729] T 82d9fd20b6c6490dacf9fc06bc8be060 P f4d5072d773d4bb59347985fb601fe31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d5072d773d4bb59347985fb601fe31, State: Initialized, Role: FOLLOWER
I20260927 15:18:00.416563 29090 raft_consensus.cc:729] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea86079091d449219270441e7075cd51, State: Initialized, Role: FOLLOWER
I20260927 15:18:00.417300 29090 consensus_queue.cc:260] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d5072d773d4bb59347985fb601fe31" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35095 } } peers { permanent_uuid: "ea86079091d449219270441e7075cd51" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34395 } } peers { permanent_uuid: "a714a4227be54aa29ca2981f2bfe4a6c" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 46725 } }
I20260927 15:18:00.417300 29091 consensus_queue.cc:260] T 82d9fd20b6c6490dacf9fc06bc8be060 P f4d5072d773d4bb59347985fb601fe31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d5072d773d4bb59347985fb601fe31" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35095 } } peers { permanent_uuid: "ea86079091d449219270441e7075cd51" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34395 } } peers { permanent_uuid: "a714a4227be54aa29ca2981f2bfe4a6c" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 46725 } }
I20260927 15:18:00.417841 29092 ts_tablet_manager.cc:1428] T 82d9fd20b6c6490dacf9fc06bc8be060 P a714a4227be54aa29ca2981f2bfe4a6c: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260927 15:18:00.419258 29072 heartbeater.cc:503] Master 127.10.33.126:40025 was elected leader, sending a full tablet report...
I20260927 15:18:00.419500 29091 ts_tablet_manager.cc:1428] T 82d9fd20b6c6490dacf9fc06bc8be060 P f4d5072d773d4bb59347985fb601fe31: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260927 15:18:00.419847 29090 ts_tablet_manager.cc:1428] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 15:18:00.420035 28938 heartbeater.cc:503] Master 127.10.33.126:40025 was elected leader, sending a full tablet report...
W20260927 15:18:00.491366 28805 tablet.cc:2369] T 82d9fd20b6c6490dacf9fc06bc8be060 P a714a4227be54aa29ca2981f2bfe4a6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:18:00.501821 29098 raft_consensus.cc:482] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:18:00.502175 29098 raft_consensus.cc:504] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d5072d773d4bb59347985fb601fe31" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35095 } } peers { permanent_uuid: "ea86079091d449219270441e7075cd51" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34395 } } peers { permanent_uuid: "a714a4227be54aa29ca2981f2bfe4a6c" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 46725 } }
I20260927 15:18:00.503937 29098 leader_election.cc:288] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4d5072d773d4bb59347985fb601fe31 (127.10.33.66:35095), a714a4227be54aa29ca2981f2bfe4a6c (127.10.33.65:46725)
I20260927 15:18:00.510604 29074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.504343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:35095 (local address 127.10.33.67:39221)
0927 15:18:00.504548 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:00.504568 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:00.504713 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:00.505232 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:00.505237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:00.505268 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:00.505438 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:00.505447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.506942 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.506948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:00.508425 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:00.508438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.509821 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.509827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:00.510050 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:00.510086 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:18:00.510190 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 15:18:00.510342 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 15:18:00.510629 29085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.504444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:39221 (local address 127.10.33.66:35095)
0927 15:18:00.504615 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:00.504620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:00.504639 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:18:00.504872 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:00.504878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:00.504959 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:00.505068 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:00.505598 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.506796 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.508647 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.509650 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.509771 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:00.509827 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:00.510223 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 15:18:00.510355 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:18:00.511204 29099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.504761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:46725 (local address 127.10.33.67:58057)
0927 15:18:00.505435 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:00.505449 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:00.505583 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:00.505901 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:00.505906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:00.505930 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:00.506303 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:00.506314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.507484 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.507488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:00.509057 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:00.509072 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.510386 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.510391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:00.510571 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:00.510607 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:18:00.510718 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 15:18:00.510863 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":363,"threads_started":1}
I20260927 15:18:00.511420 29089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.504886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:58057 (local address 127.10.33.65:46725)
0927 15:18:00.505115 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:00.505120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:00.505542 (+ 422us) server_negotiation.cc:407] Connection header received
0927 15:18:00.505709 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:00.505713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:00.505772 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:00.505840 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:00.506468 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.507317 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.509224 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.510187 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.510329 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:00.510396 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:00.511060 (+ 664us) server_negotiation.cc:299] Negotiation successful
0927 15:18:00.511206 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:18:00.511710 28893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d9fd20b6c6490dacf9fc06bc8be060" candidate_uuid: "ea86079091d449219270441e7075cd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d5072d773d4bb59347985fb601fe31" is_pre_election: true
I20260927 15:18:00.512248 28893 raft_consensus.cc:2454] T 82d9fd20b6c6490dacf9fc06bc8be060 P f4d5072d773d4bb59347985fb601fe31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea86079091d449219270441e7075cd51 in term 0.
I20260927 15:18:00.512387 28759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d9fd20b6c6490dacf9fc06bc8be060" candidate_uuid: "ea86079091d449219270441e7075cd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a714a4227be54aa29ca2981f2bfe4a6c" is_pre_election: true
I20260927 15:18:00.512909 28759 raft_consensus.cc:2454] T 82d9fd20b6c6490dacf9fc06bc8be060 P a714a4227be54aa29ca2981f2bfe4a6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea86079091d449219270441e7075cd51 in term 0.
I20260927 15:18:00.513054 28960 leader_election.cc:302] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea86079091d449219270441e7075cd51, f4d5072d773d4bb59347985fb601fe31; no voters:
I20260927 15:18:00.513487 29098 raft_consensus.cc:2781] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:18:00.513615 29098 raft_consensus.cc:482] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:18:00.513718 29098 raft_consensus.cc:3037] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:00.515980 29098 raft_consensus.cc:504] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d5072d773d4bb59347985fb601fe31" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35095 } } peers { permanent_uuid: "ea86079091d449219270441e7075cd51" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34395 } } peers { permanent_uuid: "a714a4227be54aa29ca2981f2bfe4a6c" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 46725 } }
I20260927 15:18:00.516588 29098 leader_election.cc:288] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51 [CANDIDATE]: Term 1 election: Requested vote from peers f4d5072d773d4bb59347985fb601fe31 (127.10.33.66:35095), a714a4227be54aa29ca2981f2bfe4a6c (127.10.33.65:46725)
I20260927 15:18:00.517056 28893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d9fd20b6c6490dacf9fc06bc8be060" candidate_uuid: "ea86079091d449219270441e7075cd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d5072d773d4bb59347985fb601fe31"
I20260927 15:18:00.517194 28759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d9fd20b6c6490dacf9fc06bc8be060" candidate_uuid: "ea86079091d449219270441e7075cd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a714a4227be54aa29ca2981f2bfe4a6c"
I20260927 15:18:00.517300 28893 raft_consensus.cc:3037] T 82d9fd20b6c6490dacf9fc06bc8be060 P f4d5072d773d4bb59347985fb601fe31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:00.517397 28759 raft_consensus.cc:3037] T 82d9fd20b6c6490dacf9fc06bc8be060 P a714a4227be54aa29ca2981f2bfe4a6c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:00.520572 28893 raft_consensus.cc:2454] T 82d9fd20b6c6490dacf9fc06bc8be060 P f4d5072d773d4bb59347985fb601fe31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea86079091d449219270441e7075cd51 in term 1.
I20260927 15:18:00.520572 28759 raft_consensus.cc:2454] T 82d9fd20b6c6490dacf9fc06bc8be060 P a714a4227be54aa29ca2981f2bfe4a6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea86079091d449219270441e7075cd51 in term 1.
I20260927 15:18:00.521049 28960 leader_election.cc:302] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea86079091d449219270441e7075cd51, f4d5072d773d4bb59347985fb601fe31; no voters:
I20260927 15:18:00.521360 29098 raft_consensus.cc:2781] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:00.521790 29098 raft_consensus.cc:686] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51 [term 1 LEADER]: Becoming Leader. State: Replica: ea86079091d449219270441e7075cd51, State: Running, Role: LEADER
I20260927 15:18:00.522323 29098 consensus_queue.cc:237] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d5072d773d4bb59347985fb601fe31" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35095 } } peers { permanent_uuid: "ea86079091d449219270441e7075cd51" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34395 } } peers { permanent_uuid: "a714a4227be54aa29ca2981f2bfe4a6c" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 46725 } }
W20260927 15:18:00.524755 28939 tablet.cc:2369] T 82d9fd20b6c6490dacf9fc06bc8be060 P f4d5072d773d4bb59347985fb601fe31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:18:00.527132 28614 catalog_manager.cc:5495] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea86079091d449219270441e7075cd51 (127.10.33.67). New cstate: current_term: 1 leader_uuid: "ea86079091d449219270441e7075cd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d5072d773d4bb59347985fb601fe31" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea86079091d449219270441e7075cd51" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a714a4227be54aa29ca2981f2bfe4a6c" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 46725 } health_report { overall_health: UNKNOWN } } }
W20260927 15:18:00.535651 29073 tablet.cc:2369] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:18:00.576583 29087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.571202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36602 (local address 127.10.33.67:34395)
0927 15:18:00.571426 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:00.571432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:00.571455 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:18:00.571612 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:00.571617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:00.571696 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:00.571820 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:00.572191 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.572885 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.574396 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.574832 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.574930 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:00.575438 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:00.575959 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:00.576096 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:00.576254 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 15:18:00.576312 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:18:00.584939 28893 raft_consensus.cc:1261] T 82d9fd20b6c6490dacf9fc06bc8be060 P f4d5072d773d4bb59347985fb601fe31 [term 1 FOLLOWER]: Refusing update from remote peer ea86079091d449219270441e7075cd51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:18:00.584940 28759 raft_consensus.cc:1261] T 82d9fd20b6c6490dacf9fc06bc8be060 P a714a4227be54aa29ca2981f2bfe4a6c [term 1 FOLLOWER]: Refusing update from remote peer ea86079091d449219270441e7075cd51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:18:00.585935 29098 consensus_queue.cc:1035] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a714a4227be54aa29ca2981f2bfe4a6c" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 46725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:18:00.586325 29100 consensus_queue.cc:1035] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d5072d773d4bb59347985fb601fe31" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 35095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:18:00.603261 29103 mvcc.cc:205] Tried to move back new op lower bound from 7333979261265510400 to 7333979261023371264. Current Snapshot: MvccSnapshot[applied={T|T < 7333979261265510400 or (T in {7333979261265510400})}]
I20260927 15:18:00.605311 29104 mvcc.cc:205] Tried to move back new op lower bound from 7333979261265510400 to 7333979261023371264. Current Snapshot: MvccSnapshot[applied={T|T < 7333979261265510400}]
I20260927 15:18:00.615834 28614 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:18:00.616132 28614 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:18:00.620168 28614 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6a1b80eaa9743818a6a832c6cf97c54: Sending DeleteTablet for 3 replicas of tablet 82d9fd20b6c6490dacf9fc06bc8be060
I20260927 15:18:00.621410 28873 tablet_service.cc:1514] Processing DeleteTablet for tablet 82d9fd20b6c6490dacf9fc06bc8be060 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:18:00 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:56902
I20260927 15:18:00.621568 28739 tablet_service.cc:1514] Processing DeleteTablet for tablet 82d9fd20b6c6490dacf9fc06bc8be060 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:18:00 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:57586
I20260927 15:18:00.621585 29007 tablet_service.cc:1514] Processing DeleteTablet for tablet 82d9fd20b6c6490dacf9fc06bc8be060 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:18:00 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:36598
I20260927 15:18:00.625891 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 28676
I20260927 15:18:00.639258 29114 tablet_replica.cc:330] T 82d9fd20b6c6490dacf9fc06bc8be060 P f4d5072d773d4bb59347985fb601fe31: stopping tablet replica
I20260927 15:18:00.640126 29114 raft_consensus.cc:2229] T 82d9fd20b6c6490dacf9fc06bc8be060 P f4d5072d773d4bb59347985fb601fe31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:18:00.640920 29114 raft_consensus.cc:2258] T 82d9fd20b6c6490dacf9fc06bc8be060 P f4d5072d773d4bb59347985fb601fe31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:18:00.642978 29113 tablet_replica.cc:330] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51: stopping tablet replica
I20260927 15:18:00.643605 29114 ts_tablet_manager.cc:1905] T 82d9fd20b6c6490dacf9fc06bc8be060 P f4d5072d773d4bb59347985fb601fe31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:18:00.644100 29113 raft_consensus.cc:2229] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:18:00.644196 29112 tablet_replica.cc:330] T 82d9fd20b6c6490dacf9fc06bc8be060 P a714a4227be54aa29ca2981f2bfe4a6c: stopping tablet replica
I20260927 15:18:00.644608 29112 raft_consensus.cc:2229] T 82d9fd20b6c6490dacf9fc06bc8be060 P a714a4227be54aa29ca2981f2bfe4a6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:18:00.645007 29113 raft_consensus.cc:2258] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:18:00.645155 29112 raft_consensus.cc:2258] T 82d9fd20b6c6490dacf9fc06bc8be060 P a714a4227be54aa29ca2981f2bfe4a6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:18:00.647765 29113 ts_tablet_manager.cc:1905] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 15:18:00 dist-test-slave-f5h6 krb5kdc[28570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.65@KRBTEST.COM
I20260927 15:18:00.652737 29114 ts_tablet_manager.cc:1918] T 82d9fd20b6c6490dacf9fc06bc8be060 P f4d5072d773d4bb59347985fb601fe31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:18:00.653192 29114 log.cc:1196] T 82d9fd20b6c6490dacf9fc06bc8be060 P f4d5072d773d4bb59347985fb601fe31: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/82d9fd20b6c6490dacf9fc06bc8be060
I20260927 15:18:00.653843 29114 ts_tablet_manager.cc:1939] T 82d9fd20b6c6490dacf9fc06bc8be060 P f4d5072d773d4bb59347985fb601fe31: Deleting consensus metadata
I20260927 15:18:00.655282 29112 ts_tablet_manager.cc:1905] T 82d9fd20b6c6490dacf9fc06bc8be060 P a714a4227be54aa29ca2981f2bfe4a6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:18:00.656095 28599 catalog_manager.cc:4841] TS f4d5072d773d4bb59347985fb601fe31 (127.10.33.66:35095): tablet 82d9fd20b6c6490dacf9fc06bc8be060 (table test-table [id=7b4c69f6864a45eca33a6fb0fb3c1d2a]) successfully deleted
I20260927 15:18:00.658345 29089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.626790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57588 (local address 127.10.33.65:46725)
0927 15:18:00.626979 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:00.626986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:00.632922 (+ 5936us) server_negotiation.cc:407] Connection header received
0927 15:18:00.634277 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:00.634285 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:00.634389 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:00.634473 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:00.638588 (+ 4115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.640074 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.642816 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.643522 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.643634 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:00.649595 (+ 5961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:00.649618 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:00.649639 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:00.649692 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:00.654082 (+ 4390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:00.655283 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:00.655294 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:00.655297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:00.655526 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:00.656046 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:00.656052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:00.656055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:00.656615 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:00.656943 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:00.657628 (+ 685us) server_negotiation.cc:299] Negotiation successful
0927 15:18:00.657934 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:18:00.659359 29113 ts_tablet_manager.cc:1918] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:18:00.659631 29113 log.cc:1196] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/82d9fd20b6c6490dacf9fc06bc8be060
I20260927 15:18:00.660207 28799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:00.660264 29113 ts_tablet_manager.cc:1939] T 82d9fd20b6c6490dacf9fc06bc8be060 P ea86079091d449219270441e7075cd51: Deleting consensus metadata
I20260927 15:18:00.661919 28599 catalog_manager.cc:4841] TS ea86079091d449219270441e7075cd51 (127.10.33.67:34395): tablet 82d9fd20b6c6490dacf9fc06bc8be060 (table test-table [id=7b4c69f6864a45eca33a6fb0fb3c1d2a]) successfully deleted
I20260927 15:18:00.752907 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 28676
I20260927 15:18:00.767868 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 28810
W20260927 15:18:00.767922 28599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.33.65:46725 (error 108)
I20260927 15:18:00.770164 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.769544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:46725 (local address 127.0.0.1:57602)
0927 15:18:00.769780 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:00.769972 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260927 15:18:00.770418 28599 catalog_manager.cc:4584] TS a714a4227be54aa29ca2981f2bfe4a6c (127.10.33.65:46725): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82d9fd20b6c6490dacf9fc06bc8be060: Network error: Client connection negotiation failed: client connection to 127.10.33.65:46725: connect: Connection refused (error 111)
Sep 27 15:18:00 dist-test-slave-f5h6 krb5kdc[28570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.66@KRBTEST.COM
I20260927 15:18:00.788175 29085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.769013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56906 (local address 127.10.33.66:35095)
0927 15:18:00.769190 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:00.769196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:00.769218 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:18:00.769373 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:00.769377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:00.769442 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:00.769545 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:00.771342 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.772519 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.774422 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.775085 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.775220 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:00.780640 (+ 5420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:00.780655 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:00.780674 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:00.780723 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:00.784603 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:00.785607 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:00.785617 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:00.785621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:00.785790 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:00.786344 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:00.786349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:00.786353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:00.786894 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:00.787218 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:00.787547 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 15:18:00.787795 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:18:00.788857 28933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:00.823976 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.823462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:46725 (local address 127.0.0.1:57608)
0927 15:18:00.823672 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:00.823772 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 15:18:00.825032 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.824663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:46725 (local address 127.0.0.1:57618)
0927 15:18:00.824827 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:00.824913 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 15:18:00.886018 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.885424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:46725 (local address 127.0.0.1:57628)
0927 15:18:00.885674 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:00.885788 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 15:18:00.887248 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.886810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:46725 (local address 127.0.0.1:57638)
0927 15:18:00.886979 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:00.887100 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 15:18:00.894958 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 28810
I20260927 15:18:00.909229 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 28944
Sep 27 15:18:00 dist-test-slave-f5h6 krb5kdc[28570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.67@KRBTEST.COM
I20260927 15:18:00.929735 29087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.910055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.10.33.67:34395)
0927 15:18:00.910250 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:00.910255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:00.910276 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:18:00.910464 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:00.910470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:00.910537 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:00.910651 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:00.911908 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.912738 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.914118 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:00.914608 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:00.914740 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:00.923256 (+ 8516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:00.923272 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:00.923288 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:00.923333 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:00.926727 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:00.927554 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:00.927562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:00.927564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:00.927743 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:00.928128 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:00.928132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:00.928134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:00.928526 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:00.928862 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:00.929155 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 15:18:00.929376 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 15:18:00.930540 29067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:00.968772 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.968182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:46725 (local address 127.0.0.1:57648)
0927 15:18:00.968440 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:00.968563 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260927 15:18:00.970022 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:00.969569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:46725 (local address 127.0.0.1:57660)
0927 15:18:00.969771 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:00.969843 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 15:18:01.067672 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 28944
I20260927 15:18:01.084378 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 28583
I20260927 15:18:01.085601 28645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:01.197783 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:01.196924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:46725 (local address 127.0.0.1:57670)
0927 15:18:01.197115 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:01.197227 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 15:18:01.198199 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 28583
2026-09-27T15:18:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4350 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T15:18:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:01Z Disabled control of system clock
I20260927 15:18:01.270968 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:37569
--webserver_interface=127.10.33.126
--webserver_port=0
--builtin_ntp_servers=127.10.33.84:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:37569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:18:01.614809 29124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:01.615173 29124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:01.615264 29124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:01.623351 29124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:01.623495 29124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:01.623534 29124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:01.623585 29124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:01.623636 29124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:01.633571 29124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:37569
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:37569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:01.635975 29124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:01.638294 29124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:01.650974 29129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:01.652678 29132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:01.652693 29130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:01.653281 29124 server_base.cc:1034] running on GCE node
I20260927 15:18:01.654196 29124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:01.655839 29124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:01.657117 29124 hybrid_clock.cc:648] HybridClock initialized: now 1790522281657063 us; error 55 us; skew 500 ppm
I20260927 15:18:01.657500 29124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:18:01.660308 29124 webserver.cc:460] Webserver started at http://127.10.33.126:41499/ using document root <none> and password file <none>
I20260927 15:18:01.661057 29124 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:01.661172 29124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:01.661482 29124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:01.664629 29124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "9230cf0e611242af8a06126ca2d44e85"
format_stamp: "Formatted at 2026-09-27 15:18:01 on dist-test-slave-f5h6"
server_key: "3892407f543f725a44c4d235f3b81240"
server_key_iv: "2426597ae8a3d2ebcea38bac3315ab2b"
server_key_version: "encryptionkey@0"
I20260927 15:18:01.665493 29124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "9230cf0e611242af8a06126ca2d44e85"
format_stamp: "Formatted at 2026-09-27 15:18:01 on dist-test-slave-f5h6"
server_key: "3892407f543f725a44c4d235f3b81240"
server_key_iv: "2426597ae8a3d2ebcea38bac3315ab2b"
server_key_version: "encryptionkey@0"
I20260927 15:18:01.671303 29124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 15:18:01.675470 29138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:01.677031 29124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:18:01.677222 29124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "9230cf0e611242af8a06126ca2d44e85"
format_stamp: "Formatted at 2026-09-27 15:18:01 on dist-test-slave-f5h6"
server_key: "3892407f543f725a44c4d235f3b81240"
server_key_iv: "2426597ae8a3d2ebcea38bac3315ab2b"
server_key_version: "encryptionkey@0"
I20260927 15:18:01.677441 29124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:01.697935 29124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:01.698886 29124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:01.699159 29124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:01.717298 29124 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:37569
I20260927 15:18:01.717322 29189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:37569 every 8 connection(s)
I20260927 15:18:01.718700 29124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:01.722709 29190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:01.727030 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 29124
I20260927 15:18:01.727250 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:01.727701 10373 external_mini_cluster.cc:1428] Setting key 12b86a557e1558706eeef81fd992386a
I20260927 15:18:01.734551 29190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85: Bootstrap starting.
I20260927 15:18:01.738238 29190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:01.739336 29190 log.cc:824] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:01.742691 29193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:01.729659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37946 (local address 127.10.33.126:37569)
0927 15:18:01.730737 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:01.730749 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:01.730807 (+ 58us) server_negotiation.cc:407] Connection header received
0927 15:18:01.733085 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:01.733115 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:01.733645 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:01.734091 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:01.734655 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:01.736173 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:01.737555 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:01.738320 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:01.738494 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:01.739740 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:01.739751 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:01.739771 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:01.740484 (+ 713us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:01.740585 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:01.740607 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:01.740705 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:01.740982 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 15:18:01.741332 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":378,"threads_started":1}
I20260927 15:18:01.743085 29190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85: No bootstrap required, opened a new log
I20260927 15:18:01.752216 29190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9230cf0e611242af8a06126ca2d44e85" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 37569 } }
I20260927 15:18:01.752734 29190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:01.752856 29190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9230cf0e611242af8a06126ca2d44e85, State: Initialized, Role: FOLLOWER
I20260927 15:18:01.753530 29190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9230cf0e611242af8a06126ca2d44e85" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 37569 } }
I20260927 15:18:01.753785 29190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:18:01.753980 29190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:18:01.754236 29190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:01.756634 29190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9230cf0e611242af8a06126ca2d44e85" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 37569 } }
I20260927 15:18:01.757246 29190 leader_election.cc:302] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9230cf0e611242af8a06126ca2d44e85; no voters:
I20260927 15:18:01.757992 29190 leader_election.cc:288] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:18:01.758219 29195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:01.758931 29195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85 [term 1 LEADER]: Becoming Leader. State: Replica: 9230cf0e611242af8a06126ca2d44e85, State: Running, Role: LEADER
I20260927 15:18:01.759524 29195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9230cf0e611242af8a06126ca2d44e85" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 37569 } }
I20260927 15:18:01.760164 29190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:01.762759 29197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9230cf0e611242af8a06126ca2d44e85. Latest consensus state: current_term: 1 leader_uuid: "9230cf0e611242af8a06126ca2d44e85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9230cf0e611242af8a06126ca2d44e85" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 37569 } } }
I20260927 15:18:01.763206 29197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:01.763429 29196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9230cf0e611242af8a06126ca2d44e85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9230cf0e611242af8a06126ca2d44e85" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 37569 } } }
I20260927 15:18:01.763855 29196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:01.767170 29203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:01.773036 29203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:01.782790 29203 catalog_manager.cc:1347] Generated new cluster ID: ea31367e438b4b289e116643f43c1852
I20260927 15:18:01.782909 29203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:01.808609 29203 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:01.809995 29203 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:01.822252 29203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85: Generated new TSK 0
I20260927 15:18:01.823311 29203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:18:01.843241 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:37569
--builtin_ntp_servers=127.10.33.84:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:18:02.191985 29214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:02.192282 29214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:02.192368 29214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:02.199604 29214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:02.199716 29214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:02.199855 29214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:18:02.209182 29214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:37569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:02.210513 29214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:02.212532 29214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:02.226187 29222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:02.226203 29220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:02.226240 29219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:02.227154 29214 server_base.cc:1034] running on GCE node
I20260927 15:18:02.227828 29214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:02.228833 29214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:02.230123 29214 hybrid_clock.cc:648] HybridClock initialized: now 1790522282230073 us; error 34 us; skew 500 ppm
I20260927 15:18:02.230479 29214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:18:02.232908 29214 webserver.cc:460] Webserver started at http://127.10.33.65:33269/ using document root <none> and password file <none>
I20260927 15:18:02.233601 29214 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:02.233708 29214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:02.234066 29214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:02.237120 29214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "3fb0a8d7597949a1a51b2fcca68c1eea"
format_stamp: "Formatted at 2026-09-27 15:18:02 on dist-test-slave-f5h6"
server_key: "e44914ca71d4ba09d989c55d8ec516dc"
server_key_iv: "0482bbd3690698a0073fe0ee77e75399"
server_key_version: "encryptionkey@0"
I20260927 15:18:02.238041 29214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "3fb0a8d7597949a1a51b2fcca68c1eea"
format_stamp: "Formatted at 2026-09-27 15:18:02 on dist-test-slave-f5h6"
server_key: "e44914ca71d4ba09d989c55d8ec516dc"
server_key_iv: "0482bbd3690698a0073fe0ee77e75399"
server_key_version: "encryptionkey@0"
I20260927 15:18:02.243710 29214 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:18:02.247262 29228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:02.248550 29214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:18:02.248721 29214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "3fb0a8d7597949a1a51b2fcca68c1eea"
format_stamp: "Formatted at 2026-09-27 15:18:02 on dist-test-slave-f5h6"
server_key: "e44914ca71d4ba09d989c55d8ec516dc"
server_key_iv: "0482bbd3690698a0073fe0ee77e75399"
server_key_version: "encryptionkey@0"
I20260927 15:18:02.248919 29214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:02.280103 29214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:02.280982 29214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:02.281271 29214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:02.282341 29214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:18:02.283787 29214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:18:02.283885 29214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:02.284003 29214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:18:02.284070 29214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:02.315482 29214 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:46085
I20260927 15:18:02.315560 29340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:46085 every 8 connection(s)
I20260927 15:18:02.316815 29214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:18:02.324249 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 29214
I20260927 15:18:02.324412 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:18:02.324841 10373 external_mini_cluster.cc:1428] Setting key ce633ee05bfe9023f3a3ef77a4ef3cf6
I20260927 15:18:02.327888 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:37569
--builtin_ntp_servers=127.10.33.84:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:18:02.330926 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:02.319695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:59079 (local address 127.10.33.126:37569)
0927 15:18:02.320175 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:02.320180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:02.321624 (+ 1444us) server_negotiation.cc:407] Connection header received
0927 15:18:02.323582 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:02.323589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:02.323687 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:02.323782 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:02.325163 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:02.326096 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:02.328041 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:02.328459 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:02.328579 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:02.330043 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:02.330048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:02.330051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:02.330071 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:02.330109 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:02.330121 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:02.330210 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:02.330436 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 15:18:02.330543 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":239,"threads_started":1}
I20260927 15:18:02.332031 29344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:02.320301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:37569 (local address 127.10.33.65:59079)
0927 15:18:02.321441 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:02.321490 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:02.323254 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:02.324105 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:02.324116 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:02.324222 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:02.324892 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:02.324911 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:02.326276 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:02.326282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:02.327883 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:02.327899 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:02.328719 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:02.328728 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:02.328950 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:02.329685 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:18:02.329688 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:02.329732 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:18:02.329825 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:02.330226 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:02.330238 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:02.330247 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 15:18:02.330413 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 15:18:02.330863 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":325,"threads_started":1}
I20260927 15:18:02.333925 29341 heartbeater.cc:347] Connected to a master server at 127.10.33.126:37569
I20260927 15:18:02.334295 29341 heartbeater.cc:464] Registering TS with master...
I20260927 15:18:02.335196 29341 heartbeater.cc:511] Master 127.10.33.126:37569 requested a full tablet report, sending...
I20260927 15:18:02.337631 29155 ts_manager.cc:194] Registered new tserver with Master: 3fb0a8d7597949a1a51b2fcca68c1eea (127.10.33.65:46085)
I20260927 15:18:02.339136 29155 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.33.65:59079
W20260927 15:18:02.655580 29345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:02.655894 29345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:02.656018 29345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:02.663439 29345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:02.663560 29345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:02.663697 29345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:18:02.673146 29345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:37569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:02.674513 29345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:02.676321 29345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:02.689316 29350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:02.689337 29353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:02.689272 29351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:02.690397 29345 server_base.cc:1034] running on GCE node
I20260927 15:18:02.691103 29345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:02.692145 29345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:02.693374 29345 hybrid_clock.cc:648] HybridClock initialized: now 1790522282693299 us; error 66 us; skew 500 ppm
I20260927 15:18:02.693722 29345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:18:02.696213 29345 webserver.cc:460] Webserver started at http://127.10.33.66:46101/ using document root <none> and password file <none>
I20260927 15:18:02.696914 29345 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:02.697017 29345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:02.697328 29345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:02.700379 29345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "5fb5862c550c494296fd4a9209d9b631"
format_stamp: "Formatted at 2026-09-27 15:18:02 on dist-test-slave-f5h6"
server_key: "cae3d13627ef319490fd684f94e30afa"
server_key_iv: "c3836a6f70478f73da4c87b668dee61a"
server_key_version: "encryptionkey@0"
I20260927 15:18:02.701280 29345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "5fb5862c550c494296fd4a9209d9b631"
format_stamp: "Formatted at 2026-09-27 15:18:02 on dist-test-slave-f5h6"
server_key: "cae3d13627ef319490fd684f94e30afa"
server_key_iv: "c3836a6f70478f73da4c87b668dee61a"
server_key_version: "encryptionkey@0"
I20260927 15:18:02.706604 29345 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 15:18:02.710182 29359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:02.711553 29345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:18:02.711705 29345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "5fb5862c550c494296fd4a9209d9b631"
format_stamp: "Formatted at 2026-09-27 15:18:02 on dist-test-slave-f5h6"
server_key: "cae3d13627ef319490fd684f94e30afa"
server_key_iv: "c3836a6f70478f73da4c87b668dee61a"
server_key_version: "encryptionkey@0"
I20260927 15:18:02.711905 29345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:02.732954 29345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:02.733796 29345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:02.734120 29345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:02.735105 29345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:18:02.736608 29345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:18:02.736701 29345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:02.736804 29345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:18:02.736869 29345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:02.767284 29345 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:41469
I20260927 15:18:02.767349 29471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:41469 every 8 connection(s)
I20260927 15:18:02.768626 29345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:18:02.774271 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 29345
I20260927 15:18:02.774430 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:18:02.774837 10373 external_mini_cluster.cc:1428] Setting key e0c9fb1c0dc51bbebad74265bec920d0
I20260927 15:18:02.777422 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:37569
--builtin_ntp_servers=127.10.33.84:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:18:02.781404 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:02.771323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:56417 (local address 127.10.33.126:37569)
0927 15:18:02.771577 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:02.771582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:02.773004 (+ 1422us) server_negotiation.cc:407] Connection header received
0927 15:18:02.774988 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:02.774993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:02.775078 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:02.775178 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:02.776265 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:02.776966 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:02.778649 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:02.779047 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:02.779176 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:02.780561 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:02.780568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:02.780572 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:02.780596 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:02.780635 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:02.780650 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:02.780728 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:02.780945 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 15:18:02.781059 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260927 15:18:02.782584 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:02.771919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:37569 (local address 127.10.33.66:56417)
0927 15:18:02.772838 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:02.772879 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:02.774718 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:02.775414 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:02.775424 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:02.775515 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:02.776098 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:02.776115 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:02.777114 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:02.777118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:02.778502 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:02.778516 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:02.779282 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:02.779290 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:02.779516 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:02.780219 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:18:02.780222 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:02.780263 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:18:02.780356 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:02.780741 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:02.780754 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:02.780762 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 15:18:02.780923 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 15:18:02.781355 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":274,"threads_started":1}
I20260927 15:18:02.784246 29472 heartbeater.cc:347] Connected to a master server at 127.10.33.126:37569
I20260927 15:18:02.784564 29472 heartbeater.cc:464] Registering TS with master...
I20260927 15:18:02.785339 29472 heartbeater.cc:511] Master 127.10.33.126:37569 requested a full tablet report, sending...
I20260927 15:18:02.786845 29155 ts_manager.cc:194] Registered new tserver with Master: 5fb5862c550c494296fd4a9209d9b631 (127.10.33.66:41469)
I20260927 15:18:02.787614 29155 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.33.66:56417
W20260927 15:18:03.110414 29475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:03.110706 29475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:03.110818 29475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:03.118578 29475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:03.118698 29475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:03.118846 29475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:18:03.128191 29475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:37569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:03.129525 29475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:03.131299 29475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:03.144577 29483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:03.144590 29481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:03.144629 29480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:03.145282 29475 server_base.cc:1034] running on GCE node
I20260927 15:18:03.145929 29475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:03.146925 29475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:03.148140 29475 hybrid_clock.cc:648] HybridClock initialized: now 1790522283148089 us; error 38 us; skew 500 ppm
I20260927 15:18:03.148520 29475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:18:03.151279 29475 webserver.cc:460] Webserver started at http://127.10.33.67:38361/ using document root <none> and password file <none>
I20260927 15:18:03.152005 29475 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:03.152117 29475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:03.152434 29475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:03.155433 29475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "a677f36f56e14de5a4e680a262930da6"
format_stamp: "Formatted at 2026-09-27 15:18:03 on dist-test-slave-f5h6"
server_key: "5cae9e5060e9636d2d1441f550355b44"
server_key_iv: "e610ea3ad16839f9d78cce170d890194"
server_key_version: "encryptionkey@0"
I20260927 15:18:03.156195 29475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "a677f36f56e14de5a4e680a262930da6"
format_stamp: "Formatted at 2026-09-27 15:18:03 on dist-test-slave-f5h6"
server_key: "5cae9e5060e9636d2d1441f550355b44"
server_key_iv: "e610ea3ad16839f9d78cce170d890194"
server_key_version: "encryptionkey@0"
I20260927 15:18:03.161573 29475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 15:18:03.165210 29489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:03.166625 29475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:18:03.166770 29475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "a677f36f56e14de5a4e680a262930da6"
format_stamp: "Formatted at 2026-09-27 15:18:03 on dist-test-slave-f5h6"
server_key: "5cae9e5060e9636d2d1441f550355b44"
server_key_iv: "e610ea3ad16839f9d78cce170d890194"
server_key_version: "encryptionkey@0"
I20260927 15:18:03.166951 29475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:03.186465 29475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:03.187275 29475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:03.187537 29475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:03.188489 29475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:18:03.190004 29475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:18:03.190088 29475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:03.190168 29475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:18:03.190228 29475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:03.222163 29475 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:37727
I20260927 15:18:03.222191 29601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:37727 every 8 connection(s)
I20260927 15:18:03.223584 29475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:18:03.233194 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 29475
I20260927 15:18:03.233381 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:18:03.233937 10373 external_mini_cluster.cc:1428] Setting key 7684b47a4ac34947073e6bdf7a1f716e
I20260927 15:18:03.237229 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:03.226070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:34669 (local address 127.10.33.126:37569)
0927 15:18:03.226287 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:03.226294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:03.227704 (+ 1410us) server_negotiation.cc:407] Connection header received
0927 15:18:03.229642 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:03.229647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:03.229729 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:03.229831 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:03.231147 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.231930 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.233717 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.234403 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.234592 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:03.236304 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:03.236310 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:03.236313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:03.236332 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:03.236367 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:03.236379 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:03.236480 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:03.236722 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 15:18:03.236816 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:18:03.238456 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:03.226630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:37569 (local address 127.10.33.67:34669)
0927 15:18:03.227539 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:03.227575 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:03.229319 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:03.230238 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:03.230247 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:03.230344 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:03.230954 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:03.230972 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.232112 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.232116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:03.233531 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:03.233548 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.234751 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.234760 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:03.235029 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:03.235952 (+ 923us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:18:03.235956 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:03.236005 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:18:03.236115 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:03.236488 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:03.236499 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:03.236506 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:18:03.236700 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 15:18:03.237240 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":253,"threads_started":1}
I20260927 15:18:03.239886 29602 heartbeater.cc:347] Connected to a master server at 127.10.33.126:37569
I20260927 15:18:03.240247 29602 heartbeater.cc:464] Registering TS with master...
I20260927 15:18:03.241029 29602 heartbeater.cc:511] Master 127.10.33.126:37569 requested a full tablet report, sending...
I20260927 15:18:03.242691 29155 ts_manager.cc:194] Registered new tserver with Master: a677f36f56e14de5a4e680a262930da6 (127.10.33.67:37727)
I20260927 15:18:03.243500 29155 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.33.67:34669
I20260927 15:18:03.250219 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:18:03.260854 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:03.254586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37962 (local address 127.10.33.126:37569)
0927 15:18:03.254780 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:03.254784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:03.255193 (+ 409us) server_negotiation.cc:407] Connection header received
0927 15:18:03.255321 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:03.255325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:03.255394 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:03.255500 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:03.256029 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.256763 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.258369 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.258808 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.258905 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:03.260100 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:03.260105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:03.260108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:03.260129 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:03.260161 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:03.260174 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:03.260289 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:03.260563 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 15:18:03.260613 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 15:18:03.265229 29155 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:18:03.272622 29155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:18:03.297492 29615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:03.286301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:37727 (local address 127.0.0.1:34778)
0927 15:18:03.287103 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:03.287149 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:03.287319 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:03.288425 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:03.288433 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:03.288466 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:03.288947 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:03.288965 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.291178 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.291185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:03.293808 (+ 2623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:03.293830 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.296352 (+ 2522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.296359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:03.296643 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:03.296703 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 15:18:03.296849 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 15:18:03.297081 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":356,"threads_started":1}
I20260927 15:18:03.299427 29614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:03.286280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34778 (local address 127.10.33.67:37727)
0927 15:18:03.287147 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:03.287157 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:03.287534 (+ 377us) server_negotiation.cc:407] Connection header received
0927 15:18:03.287661 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:03.287670 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:03.287938 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:03.288110 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:03.289182 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.290979 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.294111 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.295716 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.295860 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:03.296196 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:03.298781 (+ 2585us) server_negotiation.cc:299] Negotiation successful
0927 15:18:03.299039 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":1418,"threads_started":1}
I20260927 15:18:03.307929 29616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:03.287464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:46085 (local address 127.0.0.1:41656)
0927 15:18:03.300725 (+ 13261us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:03.300741 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:03.300849 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:03.302109 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:03.302122 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:03.302149 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:03.302429 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:03.302437 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.303788 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.303794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:03.305754 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:03.305769 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.307170 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.307177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:03.307360 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:03.307393 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:18:03.307467 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 15:18:03.307602 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13117,"threads_started":1}
I20260927 15:18:03.308029 29619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:03.288134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.10.33.65:46085)
0927 15:18:03.298418 (+ 10284us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:03.298430 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:03.300983 (+ 2553us) server_negotiation.cc:407] Connection header received
0927 15:18:03.301123 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:03.301131 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:03.301445 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:03.301674 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:03.302604 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.303609 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.306018 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.306972 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.307096 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:03.307363 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:03.307554 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 15:18:03.307717 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10077,"thread_start_us":9946,"threads_started":1}
I20260927 15:18:03.312738 29615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:03.287286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:41469 (local address 127.0.0.1:33998)
0927 15:18:03.300028 (+ 12742us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:03.300054 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:03.300206 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:03.305711 (+ 5505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:03.305719 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:03.305756 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:03.306068 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:03.306079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.308717 (+ 2638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.308721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:03.310677 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:03.310698 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.311943 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.311949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:03.312122 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:03.312155 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:18:03.312236 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 15:18:03.312379 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12586,"mutex_wait_us":6,"thread_start_us":13157,"threads_started":1}
I20260927 15:18:03.312767 29618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:03.287534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33998 (local address 127.10.33.66:41469)
0927 15:18:03.304523 (+ 16989us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:03.304534 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:03.304572 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:18:03.304809 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:03.304819 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:03.305150 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:03.306291 (+ 1141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:03.306541 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.308156 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.310839 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.311781 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.311908 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:03.312174 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:03.312290 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 15:18:03.312453 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16792,"thread_start_us":17833,"threads_started":1}
I20260927 15:18:03.312758 29537 tablet_service.cc:1467] Processing CreateTablet for tablet b8e0f8c33c534b2e9f8325296381dc6e (DEFAULT_TABLE table=test-table [id=4962c2c16d8e432b990b2fe37b5daf36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:18:03.312758 29276 tablet_service.cc:1467] Processing CreateTablet for tablet b8e0f8c33c534b2e9f8325296381dc6e (DEFAULT_TABLE table=test-table [id=4962c2c16d8e432b990b2fe37b5daf36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:18:03.314296 29276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8e0f8c33c534b2e9f8325296381dc6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:03.314296 29537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8e0f8c33c534b2e9f8325296381dc6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:03.322219 29407 tablet_service.cc:1467] Processing CreateTablet for tablet b8e0f8c33c534b2e9f8325296381dc6e (DEFAULT_TABLE table=test-table [id=4962c2c16d8e432b990b2fe37b5daf36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:18:03.323788 29407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8e0f8c33c534b2e9f8325296381dc6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:03.329383 29620 tablet_bootstrap.cc:492] T b8e0f8c33c534b2e9f8325296381dc6e P a677f36f56e14de5a4e680a262930da6: Bootstrap starting.
I20260927 15:18:03.332683 29621 tablet_bootstrap.cc:492] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea: Bootstrap starting.
I20260927 15:18:03.334096 29620 tablet_bootstrap.cc:654] T b8e0f8c33c534b2e9f8325296381dc6e P a677f36f56e14de5a4e680a262930da6: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:03.335368 29620 log.cc:824] T b8e0f8c33c534b2e9f8325296381dc6e P a677f36f56e14de5a4e680a262930da6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:03.336400 29621 tablet_bootstrap.cc:654] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:03.337566 29621 log.cc:824] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:03.338083 29620 tablet_bootstrap.cc:492] T b8e0f8c33c534b2e9f8325296381dc6e P a677f36f56e14de5a4e680a262930da6: No bootstrap required, opened a new log
I20260927 15:18:03.338397 29620 ts_tablet_manager.cc:1397] T b8e0f8c33c534b2e9f8325296381dc6e P a677f36f56e14de5a4e680a262930da6: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 15:18:03.341181 29624 tablet_bootstrap.cc:492] T b8e0f8c33c534b2e9f8325296381dc6e P 5fb5862c550c494296fd4a9209d9b631: Bootstrap starting.
I20260927 15:18:03.342772 29341 heartbeater.cc:503] Master 127.10.33.126:37569 was elected leader, sending a full tablet report...
I20260927 15:18:03.343645 29621 tablet_bootstrap.cc:492] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea: No bootstrap required, opened a new log
I20260927 15:18:03.343926 29621 ts_tablet_manager.cc:1397] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260927 15:18:03.345111 29624 tablet_bootstrap.cc:654] T b8e0f8c33c534b2e9f8325296381dc6e P 5fb5862c550c494296fd4a9209d9b631: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:03.346383 29624 log.cc:824] T b8e0f8c33c534b2e9f8325296381dc6e P 5fb5862c550c494296fd4a9209d9b631: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:03.348271 29620 raft_consensus.cc:348] T b8e0f8c33c534b2e9f8325296381dc6e P a677f36f56e14de5a4e680a262930da6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a677f36f56e14de5a4e680a262930da6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 37727 } } peers { permanent_uuid: "3fb0a8d7597949a1a51b2fcca68c1eea" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 46085 } } peers { permanent_uuid: "5fb5862c550c494296fd4a9209d9b631" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 41469 } }
I20260927 15:18:03.348766 29620 raft_consensus.cc:374] T b8e0f8c33c534b2e9f8325296381dc6e P a677f36f56e14de5a4e680a262930da6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:03.348875 29620 raft_consensus.cc:729] T b8e0f8c33c534b2e9f8325296381dc6e P a677f36f56e14de5a4e680a262930da6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a677f36f56e14de5a4e680a262930da6, State: Initialized, Role: FOLLOWER
I20260927 15:18:03.349547 29620 consensus_queue.cc:260] T b8e0f8c33c534b2e9f8325296381dc6e P a677f36f56e14de5a4e680a262930da6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a677f36f56e14de5a4e680a262930da6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 37727 } } peers { permanent_uuid: "3fb0a8d7597949a1a51b2fcca68c1eea" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 46085 } } peers { permanent_uuid: "5fb5862c550c494296fd4a9209d9b631" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 41469 } }
I20260927 15:18:03.350672 29602 heartbeater.cc:503] Master 127.10.33.126:37569 was elected leader, sending a full tablet report...
I20260927 15:18:03.350912 29624 tablet_bootstrap.cc:492] T b8e0f8c33c534b2e9f8325296381dc6e P 5fb5862c550c494296fd4a9209d9b631: No bootstrap required, opened a new log
I20260927 15:18:03.351405 29620 ts_tablet_manager.cc:1428] T b8e0f8c33c534b2e9f8325296381dc6e P a677f36f56e14de5a4e680a262930da6: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 15:18:03.351671 29624 ts_tablet_manager.cc:1397] T b8e0f8c33c534b2e9f8325296381dc6e P 5fb5862c550c494296fd4a9209d9b631: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260927 15:18:03.354806 29621 raft_consensus.cc:348] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a677f36f56e14de5a4e680a262930da6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 37727 } } peers { permanent_uuid: "3fb0a8d7597949a1a51b2fcca68c1eea" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 46085 } } peers { permanent_uuid: "5fb5862c550c494296fd4a9209d9b631" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 41469 } }
I20260927 15:18:03.355269 29621 raft_consensus.cc:374] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:03.355368 29621 raft_consensus.cc:729] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb0a8d7597949a1a51b2fcca68c1eea, State: Initialized, Role: FOLLOWER
I20260927 15:18:03.356009 29621 consensus_queue.cc:260] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a677f36f56e14de5a4e680a262930da6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 37727 } } peers { permanent_uuid: "3fb0a8d7597949a1a51b2fcca68c1eea" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 46085 } } peers { permanent_uuid: "5fb5862c550c494296fd4a9209d9b631" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 41469 } }
I20260927 15:18:03.357707 29621 ts_tablet_manager.cc:1428] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260927 15:18:03.360787 29624 raft_consensus.cc:348] T b8e0f8c33c534b2e9f8325296381dc6e P 5fb5862c550c494296fd4a9209d9b631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a677f36f56e14de5a4e680a262930da6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 37727 } } peers { permanent_uuid: "3fb0a8d7597949a1a51b2fcca68c1eea" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 46085 } } peers { permanent_uuid: "5fb5862c550c494296fd4a9209d9b631" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 41469 } }
I20260927 15:18:03.361181 29624 raft_consensus.cc:374] T b8e0f8c33c534b2e9f8325296381dc6e P 5fb5862c550c494296fd4a9209d9b631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:03.361274 29624 raft_consensus.cc:729] T b8e0f8c33c534b2e9f8325296381dc6e P 5fb5862c550c494296fd4a9209d9b631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fb5862c550c494296fd4a9209d9b631, State: Initialized, Role: FOLLOWER
I20260927 15:18:03.361918 29624 consensus_queue.cc:260] T b8e0f8c33c534b2e9f8325296381dc6e P 5fb5862c550c494296fd4a9209d9b631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a677f36f56e14de5a4e680a262930da6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 37727 } } peers { permanent_uuid: "3fb0a8d7597949a1a51b2fcca68c1eea" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 46085 } } peers { permanent_uuid: "5fb5862c550c494296fd4a9209d9b631" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 41469 } }
I20260927 15:18:03.362959 29472 heartbeater.cc:503] Master 127.10.33.126:37569 was elected leader, sending a full tablet report...
I20260927 15:18:03.363859 29624 ts_tablet_manager.cc:1428] T b8e0f8c33c534b2e9f8325296381dc6e P 5fb5862c550c494296fd4a9209d9b631: Time spent starting tablet: real 0.012s user 0.004s sys 0.007s
I20260927 15:18:03.368245 29627 raft_consensus.cc:482] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:18:03.368530 29627 raft_consensus.cc:504] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a677f36f56e14de5a4e680a262930da6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 37727 } } peers { permanent_uuid: "3fb0a8d7597949a1a51b2fcca68c1eea" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 46085 } } peers { permanent_uuid: "5fb5862c550c494296fd4a9209d9b631" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 41469 } }
I20260927 15:18:03.370234 29627 leader_election.cc:288] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a677f36f56e14de5a4e680a262930da6 (127.10.33.67:37727), 5fb5862c550c494296fd4a9209d9b631 (127.10.33.66:41469)
I20260927 15:18:03.376348 29614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:03.370689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:53115 (local address 127.10.33.67:37727)
0927 15:18:03.370890 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:03.370896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:03.371265 (+ 369us) server_negotiation.cc:407] Connection header received
0927 15:18:03.371611 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:03.371617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:03.371696 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:03.371771 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:03.372255 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.372941 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.374761 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.375520 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.375670 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:03.375739 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:03.375972 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 15:18:03.376103 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:18:03.376384 29629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:03.370394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:37727 (local address 127.10.33.65:53115)
0927 15:18:03.371157 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:03.371175 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:03.371389 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:03.371795 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:03.371800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:03.371823 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:03.372128 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:03.372137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.373058 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.373062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:03.374592 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:03.374604 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.375690 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.375696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:03.375831 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:03.375858 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 15:18:03.375935 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 15:18:03.376067 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":417,"threads_started":1}
I20260927 15:18:03.376533 29618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:03.371167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:55031 (local address 127.10.33.66:41469)
0927 15:18:03.371385 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:03.371391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:03.371497 (+ 106us) server_negotiation.cc:407] Connection header received
0927 15:18:03.371726 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:03.371732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:03.371816 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:03.371905 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:03.372547 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.373221 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.374950 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.375699 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.375847 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:03.375911 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:03.376171 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 15:18:03.376293 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:18:03.376593 29630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:03.370587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:41469 (local address 127.10.33.65:55031)
0927 15:18:03.371385 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:03.371401 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:03.371536 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:03.372045 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:03.372048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:03.372075 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:03.372392 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:03.372400 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.373353 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.373358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:03.374786 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:03.374798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.375870 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.375876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:03.376021 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:03.376042 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:18:03.376126 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 15:18:03.376243 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":631,"threads_started":1}
I20260927 15:18:03.377485 29557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e0f8c33c534b2e9f8325296381dc6e" candidate_uuid: "3fb0a8d7597949a1a51b2fcca68c1eea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a677f36f56e14de5a4e680a262930da6" is_pre_election: true
I20260927 15:18:03.377579 29427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e0f8c33c534b2e9f8325296381dc6e" candidate_uuid: "3fb0a8d7597949a1a51b2fcca68c1eea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fb5862c550c494296fd4a9209d9b631" is_pre_election: true
I20260927 15:18:03.378091 29557 raft_consensus.cc:2454] T b8e0f8c33c534b2e9f8325296381dc6e P a677f36f56e14de5a4e680a262930da6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb0a8d7597949a1a51b2fcca68c1eea in term 0.
I20260927 15:18:03.378141 29427 raft_consensus.cc:2454] T b8e0f8c33c534b2e9f8325296381dc6e P 5fb5862c550c494296fd4a9209d9b631 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb0a8d7597949a1a51b2fcca68c1eea in term 0.
I20260927 15:18:03.378772 29229 leader_election.cc:302] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb0a8d7597949a1a51b2fcca68c1eea, a677f36f56e14de5a4e680a262930da6; no voters:
I20260927 15:18:03.379124 29627 raft_consensus.cc:2781] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:18:03.379268 29627 raft_consensus.cc:482] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:18:03.379390 29627 raft_consensus.cc:3037] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:03.381815 29627 raft_consensus.cc:504] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a677f36f56e14de5a4e680a262930da6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 37727 } } peers { permanent_uuid: "3fb0a8d7597949a1a51b2fcca68c1eea" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 46085 } } peers { permanent_uuid: "5fb5862c550c494296fd4a9209d9b631" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 41469 } }
I20260927 15:18:03.382468 29627 leader_election.cc:288] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea [CANDIDATE]: Term 1 election: Requested vote from peers a677f36f56e14de5a4e680a262930da6 (127.10.33.67:37727), 5fb5862c550c494296fd4a9209d9b631 (127.10.33.66:41469)
I20260927 15:18:03.382978 29557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e0f8c33c534b2e9f8325296381dc6e" candidate_uuid: "3fb0a8d7597949a1a51b2fcca68c1eea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a677f36f56e14de5a4e680a262930da6"
I20260927 15:18:03.383003 29427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e0f8c33c534b2e9f8325296381dc6e" candidate_uuid: "3fb0a8d7597949a1a51b2fcca68c1eea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fb5862c550c494296fd4a9209d9b631"
I20260927 15:18:03.383176 29557 raft_consensus.cc:3037] T b8e0f8c33c534b2e9f8325296381dc6e P a677f36f56e14de5a4e680a262930da6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:03.383198 29427 raft_consensus.cc:3037] T b8e0f8c33c534b2e9f8325296381dc6e P 5fb5862c550c494296fd4a9209d9b631 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:03.385353 29557 raft_consensus.cc:2454] T b8e0f8c33c534b2e9f8325296381dc6e P a677f36f56e14de5a4e680a262930da6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb0a8d7597949a1a51b2fcca68c1eea in term 1.
I20260927 15:18:03.385480 29427 raft_consensus.cc:2454] T b8e0f8c33c534b2e9f8325296381dc6e P 5fb5862c550c494296fd4a9209d9b631 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb0a8d7597949a1a51b2fcca68c1eea in term 1.
I20260927 15:18:03.385758 29229 leader_election.cc:302] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb0a8d7597949a1a51b2fcca68c1eea, a677f36f56e14de5a4e680a262930da6; no voters:
I20260927 15:18:03.386062 29627 raft_consensus.cc:2781] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:03.386489 29627 raft_consensus.cc:686] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea [term 1 LEADER]: Becoming Leader. State: Replica: 3fb0a8d7597949a1a51b2fcca68c1eea, State: Running, Role: LEADER
I20260927 15:18:03.386942 29627 consensus_queue.cc:237] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a677f36f56e14de5a4e680a262930da6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 37727 } } peers { permanent_uuid: "3fb0a8d7597949a1a51b2fcca68c1eea" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 46085 } } peers { permanent_uuid: "5fb5862c550c494296fd4a9209d9b631" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 41469 } }
I20260927 15:18:03.391862 29155 catalog_manager.cc:5495] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb0a8d7597949a1a51b2fcca68c1eea (127.10.33.65). New cstate: current_term: 1 leader_uuid: "3fb0a8d7597949a1a51b2fcca68c1eea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a677f36f56e14de5a4e680a262930da6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fb0a8d7597949a1a51b2fcca68c1eea" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 46085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fb5862c550c494296fd4a9209d9b631" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 41469 } health_report { overall_health: UNKNOWN } } }
I20260927 15:18:03.412276 29619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:03.406993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41660 (local address 127.10.33.65:46085)
0927 15:18:03.407199 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:03.407204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:03.407224 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:18:03.407332 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:03.407336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:03.407422 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:03.407554 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:03.407922 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.408627 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.410086 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.410489 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.410607 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:03.411202 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:03.411701 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:03.411857 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:03.411993 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 15:18:03.412039 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:18:03.420408 29427 raft_consensus.cc:1261] T b8e0f8c33c534b2e9f8325296381dc6e P 5fb5862c550c494296fd4a9209d9b631 [term 1 FOLLOWER]: Refusing update from remote peer 3fb0a8d7597949a1a51b2fcca68c1eea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:18:03.420408 29557 raft_consensus.cc:1261] T b8e0f8c33c534b2e9f8325296381dc6e P a677f36f56e14de5a4e680a262930da6 [term 1 FOLLOWER]: Refusing update from remote peer 3fb0a8d7597949a1a51b2fcca68c1eea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:18:03.421334 29631 consensus_queue.cc:1035] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fb5862c550c494296fd4a9209d9b631" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 41469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:18:03.421732 29627 consensus_queue.cc:1035] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea [LEADER]: Connected to new peer: Peer: permanent_uuid: "a677f36f56e14de5a4e680a262930da6" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:18:03.434661 29635 mvcc.cc:205] Tried to move back new op lower bound from 7333979272880152576 to 7333979272756649984. Current Snapshot: MvccSnapshot[applied={T|T < 7333979272880152576}]
I20260927 15:18:03.453050 29154 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:18:03.453342 29154 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:18:03.457134 29154 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9230cf0e611242af8a06126ca2d44e85: Sending DeleteTablet for 3 replicas of tablet b8e0f8c33c534b2e9f8325296381dc6e
I20260927 15:18:03.458343 29537 tablet_service.cc:1514] Processing DeleteTablet for tablet b8e0f8c33c534b2e9f8325296381dc6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:18:03 UTC) from {username='slave'} at 127.0.0.1:34778
I20260927 15:18:03.458451 29407 tablet_service.cc:1514] Processing DeleteTablet for tablet b8e0f8c33c534b2e9f8325296381dc6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:18:03 UTC) from {username='slave'} at 127.0.0.1:33998
I20260927 15:18:03.459182 29276 tablet_service.cc:1514] Processing DeleteTablet for tablet b8e0f8c33c534b2e9f8325296381dc6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:18:03 UTC) from {username='slave'} at 127.0.0.1:41656
I20260927 15:18:03.460927 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 29214
I20260927 15:18:03.477165 29647 tablet_replica.cc:330] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea: stopping tablet replica
I20260927 15:18:03.477758 29647 raft_consensus.cc:2229] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:18:03.480362 29647 raft_consensus.cc:2258] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:18:03.484149 29647 ts_tablet_manager.cc:1905] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:18:03.490360 29645 tablet_replica.cc:330] T b8e0f8c33c534b2e9f8325296381dc6e P a677f36f56e14de5a4e680a262930da6: stopping tablet replica
I20260927 15:18:03.491624 29645 raft_consensus.cc:2229] T b8e0f8c33c534b2e9f8325296381dc6e P a677f36f56e14de5a4e680a262930da6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:18:03.494311 29645 raft_consensus.cc:2258] T b8e0f8c33c534b2e9f8325296381dc6e P a677f36f56e14de5a4e680a262930da6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:18:03.497136 29619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:03.462064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41668 (local address 127.10.33.65:46085)
0927 15:18:03.462249 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:03.462255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:03.485838 (+ 23583us) server_negotiation.cc:407] Connection header received
0927 15:18:03.486386 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:03.486393 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:03.486539 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:03.486632 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:03.487303 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.488559 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.491901 (+ 3342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.492591 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.492696 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:03.495037 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:03.495050 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:03.495067 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:03.495898 (+ 831us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:03.495959 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:03.495986 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:03.496071 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:03.496611 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 15:18:03.496738 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:18:03.498342 29646 tablet_replica.cc:330] T b8e0f8c33c534b2e9f8325296381dc6e P 5fb5862c550c494296fd4a9209d9b631: stopping tablet replica
I20260927 15:18:03.498808 29646 raft_consensus.cc:2229] T b8e0f8c33c534b2e9f8325296381dc6e P 5fb5862c550c494296fd4a9209d9b631 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:18:03.499311 29646 raft_consensus.cc:2258] T b8e0f8c33c534b2e9f8325296381dc6e P 5fb5862c550c494296fd4a9209d9b631 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:18:03.499469 29336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:03.501200 29646 ts_tablet_manager.cc:1905] T b8e0f8c33c534b2e9f8325296381dc6e P 5fb5862c550c494296fd4a9209d9b631: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:18:03.501365 29645 ts_tablet_manager.cc:1905] T b8e0f8c33c534b2e9f8325296381dc6e P a677f36f56e14de5a4e680a262930da6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:18:03.509016 29646 ts_tablet_manager.cc:1918] T b8e0f8c33c534b2e9f8325296381dc6e P 5fb5862c550c494296fd4a9209d9b631: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:18:03.509232 29646 log.cc:1196] T b8e0f8c33c534b2e9f8325296381dc6e P 5fb5862c550c494296fd4a9209d9b631: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/b8e0f8c33c534b2e9f8325296381dc6e
I20260927 15:18:03.509550 29645 ts_tablet_manager.cc:1918] T b8e0f8c33c534b2e9f8325296381dc6e P a677f36f56e14de5a4e680a262930da6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:18:03.509752 29645 log.cc:1196] T b8e0f8c33c534b2e9f8325296381dc6e P a677f36f56e14de5a4e680a262930da6: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/b8e0f8c33c534b2e9f8325296381dc6e
I20260927 15:18:03.509825 29646 ts_tablet_manager.cc:1939] T b8e0f8c33c534b2e9f8325296381dc6e P 5fb5862c550c494296fd4a9209d9b631: Deleting consensus metadata
I20260927 15:18:03.510311 29645 ts_tablet_manager.cc:1939] T b8e0f8c33c534b2e9f8325296381dc6e P a677f36f56e14de5a4e680a262930da6: Deleting consensus metadata
I20260927 15:18:03.512149 29141 catalog_manager.cc:4841] TS 5fb5862c550c494296fd4a9209d9b631 (127.10.33.66:41469): tablet b8e0f8c33c534b2e9f8325296381dc6e (table test-table [id=4962c2c16d8e432b990b2fe37b5daf36]) successfully deleted
I20260927 15:18:03.512223 29139 catalog_manager.cc:4841] TS a677f36f56e14de5a4e680a262930da6 (127.10.33.67:37727): tablet b8e0f8c33c534b2e9f8325296381dc6e (table test-table [id=4962c2c16d8e432b990b2fe37b5daf36]) successfully deleted
I20260927 15:18:03.608825 29647 ts_tablet_manager.cc:1918] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:18:03.609140 29647 log.cc:1196] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/b8e0f8c33c534b2e9f8325296381dc6e
I20260927 15:18:03.609826 29647 ts_tablet_manager.cc:1939] T b8e0f8c33c534b2e9f8325296381dc6e P 3fb0a8d7597949a1a51b2fcca68c1eea: Deleting consensus metadata
I20260927 15:18:03.611967 29139 catalog_manager.cc:4841] TS 3fb0a8d7597949a1a51b2fcca68c1eea (127.10.33.65:46085): tablet b8e0f8c33c534b2e9f8325296381dc6e (table test-table [id=4962c2c16d8e432b990b2fe37b5daf36]) successfully deleted
I20260927 15:18:03.612069 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 29214
I20260927 15:18:03.627730 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 29345
I20260927 15:18:03.634577 29618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:03.628524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34014 (local address 127.10.33.66:41469)
0927 15:18:03.628724 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:03.628730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:03.628777 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:18:03.628984 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:03.628988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:03.629079 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:03.629165 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:03.629584 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.630420 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.631638 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.632069 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.632184 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:03.633209 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:03.633218 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:03.633233 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:03.633798 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:03.633844 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:03.633892 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:03.633987 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:03.634119 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 15:18:03.634227 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 15:18:03.635317 29467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:03.723589 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 29345
I20260927 15:18:03.743225 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 29475
I20260927 15:18:03.750391 29614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:03.744011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34782 (local address 127.10.33.67:37727)
0927 15:18:03.744205 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:03.744209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:03.744228 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:18:03.744330 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:03.744334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:03.744446 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:03.744537 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:03.745000 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.745747 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.747116 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:03.747615 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:03.747743 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:03.749057 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:03.749067 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:03.749082 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:03.749634 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:03.749682 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:03.749709 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:03.749795 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:03.750026 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 15:18:03.750143 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:18:03.751039 29597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:03.839130 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 29475
I20260927 15:18:03.853909 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 29124
I20260927 15:18:03.854948 29185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:03.963006 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 29124
2026-09-27T15:18:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2763 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:18:04 dist-test-slave-f5h6 krb5kdc[29655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:18:04 dist-test-slave-f5h6 krb5kdc[29655](info): set up 2 sockets
Sep 27 15:18:04 dist-test-slave-f5h6 krb5kdc[29655](info): commencing operation
krb5kdc: starting...
W20260927 15:18:04.985878 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:18:05 dist-test-slave-f5h6 krb5kdc[29655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:18:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:18:05.242297 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:36247
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:36247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:05.572113 29668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:05.572425 29668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:05.572539 29668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:05.579937 29668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:05.580068 29668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:05.580114 29668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:05.580170 29668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:05.580235 29668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:05.591058 29668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:36247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:36247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:05.592504 29668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:05.593879 29668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:05.606032 29673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:05.605996 29674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:05.605967 29676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:05.607290 29668 server_base.cc:1034] running on GCE node
I20260927 15:18:05.608196 29668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:05.609727 29668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:05.610975 29668 hybrid_clock.cc:648] HybridClock initialized: now 1790522285610926 us; error 68 us; skew 500 ppm
I20260927 15:18:05.611361 29668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:05 dist-test-slave-f5h6 krb5kdc[29655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522285, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:05.618654 29668 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:18:05.620281 29668 webserver.cc:460] Webserver started at http://127.10.33.126:38669/ using document root <none> and password file <none>
I20260927 15:18:05.620998 29668 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:05.621112 29668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:05.621379 29668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:05.624455 29668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "7de2e3ef43de4d4287b876b61ddfb19b"
format_stamp: "Formatted at 2026-09-27 15:18:05 on dist-test-slave-f5h6"
server_key: "26d159dac3ef68256d90c259f62142db"
server_key_iv: "2f73051e15e1302b5167dfa8e3bf0c23"
server_key_version: "encryptionkey@0"
I20260927 15:18:05.625200 29668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "7de2e3ef43de4d4287b876b61ddfb19b"
format_stamp: "Formatted at 2026-09-27 15:18:05 on dist-test-slave-f5h6"
server_key: "26d159dac3ef68256d90c259f62142db"
server_key_iv: "2f73051e15e1302b5167dfa8e3bf0c23"
server_key_version: "encryptionkey@0"
I20260927 15:18:05.630722 29668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 15:18:05.634253 29683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:05.635660 29668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:18:05.635811 29668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "7de2e3ef43de4d4287b876b61ddfb19b"
format_stamp: "Formatted at 2026-09-27 15:18:05 on dist-test-slave-f5h6"
server_key: "26d159dac3ef68256d90c259f62142db"
server_key_iv: "2f73051e15e1302b5167dfa8e3bf0c23"
server_key_version: "encryptionkey@0"
I20260927 15:18:05.635986 29668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:05.659914 29668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:05.664091 29668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:05.664390 29668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:05.682726 29668 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:36247
I20260927 15:18:05.682757 29734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:36247 every 8 connection(s)
I20260927 15:18:05.684070 29668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:05.687760 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 29668
I20260927 15:18:05.687947 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:05.687997 29735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:05.688498 10373 external_mini_cluster.cc:1428] Setting key 0cfb73f0e9c5420f47bae873dc0b68f1
I20260927 15:18:05.703447 29735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b: Bootstrap starting.
Sep 27 15:18:05 dist-test-slave-f5h6 krb5kdc[29655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:18:05.707312 29735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:05.708760 29735 log.cc:824] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:05.712589 29735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b: No bootstrap required, opened a new log
I20260927 15:18:05.714462 29738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:05.690558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59844 (local address 127.10.33.126:36247)
0927 15:18:05.691441 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:05.691453 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:05.691501 (+ 48us) server_negotiation.cc:407] Connection header received
0927 15:18:05.693073 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:05.693104 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:05.693529 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:05.693933 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:05.695793 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:05.697218 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:05.699189 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:05.700023 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:05.700200 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:05.705275 (+ 5075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:05.705288 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:05.705307 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:05.705370 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:05.709313 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:05.710334 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:05.710343 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:05.710353 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:05.710566 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:05.711017 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:05.711022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:05.711025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:05.711790 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:05.712115 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:05.712542 (+ 427us) server_negotiation.cc:299] Negotiation successful
0927 15:18:05.713012 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":361,"threads_started":1}
I20260927 15:18:05.720920 29735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de2e3ef43de4d4287b876b61ddfb19b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 36247 } }
I20260927 15:18:05.721402 29735 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:05.721483 29735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7de2e3ef43de4d4287b876b61ddfb19b, State: Initialized, Role: FOLLOWER
I20260927 15:18:05.722091 29735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de2e3ef43de4d4287b876b61ddfb19b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 36247 } }
I20260927 15:18:05.722325 29735 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:18:05.722429 29735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:18:05.722553 29735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:05.724789 29735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de2e3ef43de4d4287b876b61ddfb19b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 36247 } }
I20260927 15:18:05.725311 29735 leader_election.cc:302] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7de2e3ef43de4d4287b876b61ddfb19b; no voters:
I20260927 15:18:05.725914 29735 leader_election.cc:288] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:18:05.726096 29740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:05.726752 29740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b [term 1 LEADER]: Becoming Leader. State: Replica: 7de2e3ef43de4d4287b876b61ddfb19b, State: Running, Role: LEADER
I20260927 15:18:05.727279 29740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de2e3ef43de4d4287b876b61ddfb19b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 36247 } }
I20260927 15:18:05.727952 29735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:05.729713 29741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7de2e3ef43de4d4287b876b61ddfb19b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de2e3ef43de4d4287b876b61ddfb19b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 36247 } } }
I20260927 15:18:05.730119 29741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:05.731029 29742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7de2e3ef43de4d4287b876b61ddfb19b. Latest consensus state: current_term: 1 leader_uuid: "7de2e3ef43de4d4287b876b61ddfb19b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de2e3ef43de4d4287b876b61ddfb19b" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 36247 } } }
I20260927 15:18:05.731424 29742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:05.734304 29745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:05.740720 29745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:05.750535 29745 catalog_manager.cc:1347] Generated new cluster ID: 51ce3351ad754f479874600166fa292e
I20260927 15:18:05.750672 29745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:05.761587 29745 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:05.763193 29745 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:05.771139 29745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b: Generated new TSK 0
I20260927 15:18:05.772349 29745 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.33.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
Entry for principal kudu/127.10.33.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab.
I20260927 15:18:05.837604 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:36247
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:06.175192 29761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:06.175515 29761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:06.175619 29761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:06.183183 29761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:06.183305 29761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:06.183419 29761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:18:06.193710 29761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.65.keytab
--principal=kudu/127.10.33.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:36247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:06.195021 29761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:06.196341 29761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:06.209267 29766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:06.209151 29767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:06.209131 29769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:06.210016 29761 server_base.cc:1034] running on GCE node
I20260927 15:18:06.210732 29761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:06.211680 29761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:06.212893 29761 hybrid_clock.cc:648] HybridClock initialized: now 1790522286212826 us; error 40 us; skew 500 ppm
I20260927 15:18:06.213208 29761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:06 dist-test-slave-f5h6 krb5kdc[29655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522286, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:06.216348 29761 init.cc:377] Logged in from keytab as kudu/127.10.33.65@KRBTEST.COM (short username kudu)
I20260927 15:18:06.217742 29761 webserver.cc:460] Webserver started at http://127.10.33.65:44533/ using document root <none> and password file <none>
I20260927 15:18:06.218461 29761 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:06.218585 29761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:06.218876 29761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:06.221963 29761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "6a436f04dbbc445aa2f68cd0dd81614e"
format_stamp: "Formatted at 2026-09-27 15:18:06 on dist-test-slave-f5h6"
server_key: "e1b147a646fc1043144d94145601feee"
server_key_iv: "ca637482797513bfe4773cb3531d09d9"
server_key_version: "encryptionkey@0"
I20260927 15:18:06.222721 29761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "6a436f04dbbc445aa2f68cd0dd81614e"
format_stamp: "Formatted at 2026-09-27 15:18:06 on dist-test-slave-f5h6"
server_key: "e1b147a646fc1043144d94145601feee"
server_key_iv: "ca637482797513bfe4773cb3531d09d9"
server_key_version: "encryptionkey@0"
I20260927 15:18:06.227998 29761 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260927 15:18:06.231580 29776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:06.233011 29761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:18:06.233178 29761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "6a436f04dbbc445aa2f68cd0dd81614e"
format_stamp: "Formatted at 2026-09-27 15:18:06 on dist-test-slave-f5h6"
server_key: "e1b147a646fc1043144d94145601feee"
server_key_iv: "ca637482797513bfe4773cb3531d09d9"
server_key_version: "encryptionkey@0"
I20260927 15:18:06.233387 29761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:06.247370 29761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:06.251916 29761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:06.252197 29761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:06.253171 29761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:18:06.254572 29761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:18:06.254659 29761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:06.254781 29761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:18:06.254837 29761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:06.286113 29761 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:37577
I20260927 15:18:06.286130 29888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:37577 every 8 connection(s)
I20260927 15:18:06.287564 29761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:18:06.292623 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 29761
I20260927 15:18:06.292819 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:18:06.293352 10373 external_mini_cluster.cc:1428] Setting key cb9b6d8c6cd63a693e67be3e7c2bd4c4
Sep 27 15:18:06 dist-test-slave-f5h6 krb5kdc[29655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522286, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.65@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:18:06.311820 29891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:06.290503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:35609 (local address 127.10.33.126:36247)
0927 15:18:06.291025 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:06.291030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:06.292031 (+ 1001us) server_negotiation.cc:407] Connection header received
0927 15:18:06.294520 (+ 2489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:06.294526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:06.294664 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:06.294796 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:06.296984 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:06.297751 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:06.299924 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:06.300534 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:06.300663 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:06.304945 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:06.304952 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:06.304958 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:06.305005 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:06.308199 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:06.309229 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:06.309235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:06.309238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:06.309382 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:06.309917 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:06.309922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:06.309925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:06.310264 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:06.310405 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:06.311237 (+ 832us) server_negotiation.cc:299] Negotiation successful
0927 15:18:06.311473 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":403,"thread_start_us":276,"threads_started":1}
I20260927 15:18:06.312777 29892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:06.291068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:36247 (local address 127.10.33.65:35609)
0927 15:18:06.291858 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:06.291897 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:06.294185 (+ 2288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:06.295060 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:06.295070 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:06.295829 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:06.296767 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:06.296798 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:06.297950 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:06.297956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:06.299758 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:06.299773 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:06.300771 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:06.300778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:06.301062 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:06.301903 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:06.301935 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:06.304667 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:06.308545 (+ 3878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:06.308553 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:06.308570 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:06.309075 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:06.309519 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:06.309523 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:06.309525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:06.309772 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:06.310429 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:06.310436 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:06.310830 (+ 394us) client_negotiation.cc:769] Sending connection context
0927 15:18:06.311081 (+ 251us) client_negotiation.cc:240] Negotiation successful
0927 15:18:06.311525 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":287,"threads_started":1}
I20260927 15:18:06.314373 29889 heartbeater.cc:347] Connected to a master server at 127.10.33.126:36247
I20260927 15:18:06.314747 29889 heartbeater.cc:464] Registering TS with master...
I20260927 15:18:06.315701 29889 heartbeater.cc:511] Master 127.10.33.126:36247 requested a full tablet report, sending...
I20260927 15:18:06.318437 29700 ts_manager.cc:194] Registered new tserver with Master: 6a436f04dbbc445aa2f68cd0dd81614e (127.10.33.65:37577)
I20260927 15:18:06.320626 29700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.65@KRBTEST.COM'} at 127.10.33.65:35609
WARNING: no policy specified for kudu/127.10.33.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
Entry for principal kudu/127.10.33.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab.
I20260927 15:18:06.353770 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:36247
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:06.683147 29895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:06.683447 29895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:06.683581 29895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:06.691136 29895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:06.691279 29895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:06.691439 29895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:18:06.702409 29895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.66.keytab
--principal=kudu/127.10.33.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:36247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:06.703961 29895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:06.705411 29895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:06.718364 29900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:06.719231 29901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:06.719255 29903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:06.720045 29895 server_base.cc:1034] running on GCE node
I20260927 15:18:06.720686 29895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:06.721714 29895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:06.722924 29895 hybrid_clock.cc:648] HybridClock initialized: now 1790522286722852 us; error 62 us; skew 500 ppm
I20260927 15:18:06.723233 29895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:06 dist-test-slave-f5h6 krb5kdc[29655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522286, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:06.726370 29895 init.cc:377] Logged in from keytab as kudu/127.10.33.66@KRBTEST.COM (short username kudu)
I20260927 15:18:06.727866 29895 webserver.cc:460] Webserver started at http://127.10.33.66:32853/ using document root <none> and password file <none>
I20260927 15:18:06.728585 29895 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:06.728698 29895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:06.729007 29895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:06.732030 29895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "6bc1d33fe45142a5a2440803335979a3"
format_stamp: "Formatted at 2026-09-27 15:18:06 on dist-test-slave-f5h6"
server_key: "6c2a1bab851ef4806757750a411947ba"
server_key_iv: "7c349e34d72f788141d291a8115bfeb4"
server_key_version: "encryptionkey@0"
I20260927 15:18:06.732764 29895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "6bc1d33fe45142a5a2440803335979a3"
format_stamp: "Formatted at 2026-09-27 15:18:06 on dist-test-slave-f5h6"
server_key: "6c2a1bab851ef4806757750a411947ba"
server_key_iv: "7c349e34d72f788141d291a8115bfeb4"
server_key_version: "encryptionkey@0"
I20260927 15:18:06.737826 29895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 15:18:06.741477 29910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:06.742971 29895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 15:18:06.743134 29895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "6bc1d33fe45142a5a2440803335979a3"
format_stamp: "Formatted at 2026-09-27 15:18:06 on dist-test-slave-f5h6"
server_key: "6c2a1bab851ef4806757750a411947ba"
server_key_iv: "7c349e34d72f788141d291a8115bfeb4"
server_key_version: "encryptionkey@0"
I20260927 15:18:06.743359 29895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:06.756456 29895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:06.760609 29895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:06.760879 29895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:06.761819 29895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:18:06.763175 29895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:18:06.763252 29895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:06.763370 29895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:18:06.763420 29895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:06.796027 29895 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:36983
I20260927 15:18:06.796063 30022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:36983 every 8 connection(s)
I20260927 15:18:06.797317 29895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:18:06.797588 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 29895
I20260927 15:18:06.797928 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:18:06.798679 10373 external_mini_cluster.cc:1428] Setting key 46003181af34deaa4d7d5f206b336d90
Sep 27 15:18:06 dist-test-slave-f5h6 krb5kdc[29655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522286, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.66@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:18:06.820510 29891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:06.800304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:40493 (local address 127.10.33.126:36247)
0927 15:18:06.800484 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:06.800490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:06.801958 (+ 1468us) server_negotiation.cc:407] Connection header received
0927 15:18:06.804510 (+ 2552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:06.804516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:06.804617 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:06.804713 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:06.806695 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:06.807426 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:06.809666 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:06.810161 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:06.810268 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:06.814424 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:06.814429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:06.814433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:06.814469 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:06.817073 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:06.818090 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:06.818094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:06.818106 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:06.818265 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:06.818761 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:06.818764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:06.818766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:06.819077 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:06.819199 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:06.820064 (+ 865us) server_negotiation.cc:299] Negotiation successful
0927 15:18:06.820265 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:18:06.821836 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:06.800847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:36247 (local address 127.10.33.66:40493)
0927 15:18:06.801758 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:06.801820 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:06.804148 (+ 2328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:06.804978 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:06.804989 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:06.805684 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:06.806494 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:06.806518 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:06.807583 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:06.807588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:06.809407 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:06.809423 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:06.810399 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:06.810406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:06.810690 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:06.811535 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:06.811558 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:06.814151 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:06.817237 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:06.817247 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:06.817265 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:06.817929 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:06.818372 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:06.818377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:06.818381 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:06.818650 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:06.819190 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:06.819197 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:06.819652 (+ 455us) client_negotiation.cc:769] Sending connection context
0927 15:18:06.819930 (+ 278us) client_negotiation.cc:240] Negotiation successful
0927 15:18:06.820454 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":369,"threads_started":1}
I20260927 15:18:06.823344 30023 heartbeater.cc:347] Connected to a master server at 127.10.33.126:36247
I20260927 15:18:06.823711 30023 heartbeater.cc:464] Registering TS with master...
I20260927 15:18:06.824610 30023 heartbeater.cc:511] Master 127.10.33.126:36247 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.33.67@KRBTEST.COM; defaulting to no policy
I20260927 15:18:06.826201 29700 ts_manager.cc:194] Registered new tserver with Master: 6bc1d33fe45142a5a2440803335979a3 (127.10.33.66:36983)
I20260927 15:18:06.827023 29700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.66@KRBTEST.COM'} at 127.10.33.66:40493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
Entry for principal kudu/127.10.33.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab.
I20260927 15:18:06.857782 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:36247
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:07.187788 30028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:07.188104 30028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:07.188215 30028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:07.195734 30028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:07.195863 30028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:07.196014 30028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:18:07.206944 30028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.67.keytab
--principal=kudu/127.10.33.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:36247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:07.208468 30028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:07.209821 30028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:07.222838 30033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:07.222982 30034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:07.223001 30036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:07.224208 30028 server_base.cc:1034] running on GCE node
I20260927 15:18:07.224848 30028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:07.225816 30028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:07.227066 30028 hybrid_clock.cc:648] HybridClock initialized: now 1790522287227027 us; error 91 us; skew 500 ppm
I20260927 15:18:07.227401 30028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:07 dist-test-slave-f5h6 krb5kdc[29655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522287, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:07.230762 30028 init.cc:377] Logged in from keytab as kudu/127.10.33.67@KRBTEST.COM (short username kudu)
I20260927 15:18:07.232352 30028 webserver.cc:460] Webserver started at http://127.10.33.67:46003/ using document root <none> and password file <none>
I20260927 15:18:07.233059 30028 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:07.233179 30028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:07.233474 30028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:07.236720 30028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "17d14cf2f36c420694a271115ca2acda"
format_stamp: "Formatted at 2026-09-27 15:18:07 on dist-test-slave-f5h6"
server_key: "ecbfc8f47ef797a4e7b9a7919f0dd0d5"
server_key_iv: "0c05b7a0344ac2a505b3b0445f25d1a8"
server_key_version: "encryptionkey@0"
I20260927 15:18:07.237556 30028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "17d14cf2f36c420694a271115ca2acda"
format_stamp: "Formatted at 2026-09-27 15:18:07 on dist-test-slave-f5h6"
server_key: "ecbfc8f47ef797a4e7b9a7919f0dd0d5"
server_key_iv: "0c05b7a0344ac2a505b3b0445f25d1a8"
server_key_version: "encryptionkey@0"
I20260927 15:18:07.243111 30028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:18:07.246814 30043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:07.248348 30028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 15:18:07.248510 30028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "17d14cf2f36c420694a271115ca2acda"
format_stamp: "Formatted at 2026-09-27 15:18:07 on dist-test-slave-f5h6"
server_key: "ecbfc8f47ef797a4e7b9a7919f0dd0d5"
server_key_iv: "0c05b7a0344ac2a505b3b0445f25d1a8"
server_key_version: "encryptionkey@0"
I20260927 15:18:07.248731 30028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:07.279029 30028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:07.283588 30028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:07.283885 30028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:07.284902 30028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:18:07.286372 30028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:18:07.286453 30028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:07.286563 30028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:18:07.286619 30028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:07.318257 30028 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:40231
I20260927 15:18:07.318320 30155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:40231 every 8 connection(s)
I20260927 15:18:07.319676 30028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:18:07.322713 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 30028
I20260927 15:18:07.322921 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:18:07.323524 10373 external_mini_cluster.cc:1428] Setting key c695e2de54ddbd8ecd938dbbb527faff
I20260927 15:18:07.324481 29889 heartbeater.cc:503] Master 127.10.33.126:36247 was elected leader, sending a full tablet report...
Sep 27 15:18:07 dist-test-slave-f5h6 krb5kdc[29655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522287, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.67@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:18:07.341107 30158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:07.322421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:58621 (local address 127.10.33.126:36247)
0927 15:18:07.322953 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:07.322958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:07.324574 (+ 1616us) server_negotiation.cc:407] Connection header received
0927 15:18:07.326453 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:07.326459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:07.326544 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:07.326635 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:07.328386 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.329270 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.330874 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.331278 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.331390 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:07.334906 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:07.334912 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:07.334916 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:07.334950 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:07.337778 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:07.338567 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:07.338571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:07.338573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:07.338730 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:07.339270 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:07.339274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:07.339277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:07.339599 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:07.339757 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:07.340631 (+ 874us) server_negotiation.cc:299] Negotiation successful
0927 15:18:07.340830 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":278,"threads_started":1}
I20260927 15:18:07.342093 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:07.322967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:36247 (local address 127.10.33.67:58621)
0927 15:18:07.323851 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:07.323891 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:07.326180 (+ 2289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:07.326935 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:07.326948 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:07.327591 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:07.328222 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:07.328240 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.329423 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.329427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:07.330729 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:07.330742 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.331514 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.331521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:07.331764 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:07.332410 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:07.332434 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:07.334662 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:07.337948 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:07.337958 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:07.337978 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:07.338431 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:07.338852 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:07.338884 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:07.338887 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:07.339152 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:07.339776 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:07.339785 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:07.340238 (+ 453us) client_negotiation.cc:769] Sending connection context
0927 15:18:07.340476 (+ 238us) client_negotiation.cc:240] Negotiation successful
0927 15:18:07.340911 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":368,"threads_started":1}
I20260927 15:18:07.343394 30156 heartbeater.cc:347] Connected to a master server at 127.10.33.126:36247
I20260927 15:18:07.343679 30156 heartbeater.cc:464] Registering TS with master...
I20260927 15:18:07.344343 30156 heartbeater.cc:511] Master 127.10.33.126:36247 requested a full tablet report, sending...
I20260927 15:18:07.345798 29700 ts_manager.cc:194] Registered new tserver with Master: 17d14cf2f36c420694a271115ca2acda (127.10.33.67:40231)
I20260927 15:18:07.346642 29700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.33.67@KRBTEST.COM'} at 127.10.33.67:58621
I20260927 15:18:07.352217 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:18:07.370258 30158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:07.356518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.10.33.126:36247)
0927 15:18:07.356767 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:07.356774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:07.356951 (+ 177us) server_negotiation.cc:407] Connection header received
0927 15:18:07.357186 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:07.357190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:07.357263 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:07.357429 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:07.359052 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.359932 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.361305 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.361753 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.361905 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:07.364208 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:07.364214 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:07.364218 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:07.364256 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:07.366844 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:07.367784 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:07.367788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:07.367791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:07.367931 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:07.368354 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:07.368358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:07.368361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:07.368765 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:07.368991 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:07.369513 (+ 522us) server_negotiation.cc:299] Negotiation successful
0927 15:18:07.369756 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 15:18:07.374522 29700 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:18:07.381500 29700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:18:07.416517 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:07.394946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37514 (local address 127.10.33.66:36983)
0927 15:18:07.408868 (+ 13922us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:07.408876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:07.408903 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:18:07.408989 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:07.408994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:07.409153 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:07.409319 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:07.409980 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.411431 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.413657 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.415032 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.415172 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:07.415386 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:07.416022 (+ 636us) server_negotiation.cc:299] Negotiation successful
0927 15:18:07.416207 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13736,"thread_start_us":13777,"threads_started":1}
I20260927 15:18:07.416595 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:07.395572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.10.33.67:40231)
0927 15:18:07.405544 (+ 9972us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:07.405554 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:07.407667 (+ 2113us) server_negotiation.cc:407] Connection header received
0927 15:18:07.407784 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:07.407792 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:07.408068 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:07.408266 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:07.409450 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.411069 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.414121 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.415217 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.415334 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:07.415544 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:07.416137 (+ 593us) server_negotiation.cc:299] Negotiation successful
0927 15:18:07.416283 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9765,"thread_start_us":9636,"threads_started":1}
I20260927 15:18:07.416595 30172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:07.394748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:36983 (local address 127.0.0.1:37514)
0927 15:18:07.405620 (+ 10872us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:07.405688 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:07.406715 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:07.409342 (+ 2627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:07.409346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:07.409372 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:07.409763 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:07.409776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.411589 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.411594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:07.413505 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:07.413517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.415522 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.415529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:07.415763 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:07.415826 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 15:18:07.415977 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 15:18:07.416190 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10562,"mutex_wait_us":55,"thread_start_us":12634,"threads_started":1}
I20260927 15:18:07.417222 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:07.394998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:40231 (local address 127.0.0.1:57360)
0927 15:18:07.407361 (+ 12363us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:07.407380 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:07.407527 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:07.408627 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:07.408636 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:07.408668 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:07.409217 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:07.409238 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.411244 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.411250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:07.413966 (+ 2716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:07.413976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.415700 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.415706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:07.415872 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:07.415907 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 15:18:07.415994 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 15:18:07.416869 (+ 875us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12228,"mutex_wait_us":917,"thread_start_us":12062,"threads_started":1}
I20260927 15:18:07.417541 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:07.396040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50206 (local address 127.10.33.65:37577)
0927 15:18:07.406108 (+ 10068us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:07.406116 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:07.407766 (+ 1650us) server_negotiation.cc:407] Connection header received
0927 15:18:07.408031 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:07.408038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:07.408241 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:07.408384 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:07.409613 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.410941 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.414080 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.415469 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.415600 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:07.415807 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:07.416307 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 15:18:07.416483 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9910,"thread_start_us":9682,"threads_started":1}
I20260927 15:18:07.418301 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:07.395382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:37577 (local address 127.0.0.1:50206)
0927 15:18:07.407657 (+ 12275us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:07.407673 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:07.407862 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:07.408654 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:07.408657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:07.408675 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:07.409348 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:07.409360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.411118 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.411123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:07.413905 (+ 2782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:07.413925 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.415949 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.415956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:07.416107 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:07.416136 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 15:18:07.417795 (+ 1659us) client_negotiation.cc:240] Negotiation successful
0927 15:18:07.417948 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12160,"mutex_wait_us":153,"thread_start_us":10498,"threads_started":1}
I20260927 15:18:07.420817 29958 tablet_service.cc:1467] Processing CreateTablet for tablet e211d836faf44c859f49edf707010309 (DEFAULT_TABLE table=test-table [id=76ad30a2df0f47e0a60e0cc1ace3d8e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:18:07.420924 30091 tablet_service.cc:1467] Processing CreateTablet for tablet e211d836faf44c859f49edf707010309 (DEFAULT_TABLE table=test-table [id=76ad30a2df0f47e0a60e0cc1ace3d8e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:18:07.422051 29958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e211d836faf44c859f49edf707010309. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:07.421556 29824 tablet_service.cc:1467] Processing CreateTablet for tablet e211d836faf44c859f49edf707010309 (DEFAULT_TABLE table=test-table [id=76ad30a2df0f47e0a60e0cc1ace3d8e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:18:07.422333 30091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e211d836faf44c859f49edf707010309. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:07.423257 29824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e211d836faf44c859f49edf707010309. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:07.435081 30175 tablet_bootstrap.cc:492] T e211d836faf44c859f49edf707010309 P 6bc1d33fe45142a5a2440803335979a3: Bootstrap starting.
I20260927 15:18:07.438081 30177 tablet_bootstrap.cc:492] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e: Bootstrap starting.
I20260927 15:18:07.439793 30175 tablet_bootstrap.cc:654] T e211d836faf44c859f49edf707010309 P 6bc1d33fe45142a5a2440803335979a3: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:07.441252 30175 log.cc:824] T e211d836faf44c859f49edf707010309 P 6bc1d33fe45142a5a2440803335979a3: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:07.443440 30177 tablet_bootstrap.cc:654] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:07.444797 30177 log.cc:824] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:07.444895 30175 tablet_bootstrap.cc:492] T e211d836faf44c859f49edf707010309 P 6bc1d33fe45142a5a2440803335979a3: No bootstrap required, opened a new log
I20260927 15:18:07.445173 30175 ts_tablet_manager.cc:1397] T e211d836faf44c859f49edf707010309 P 6bc1d33fe45142a5a2440803335979a3: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 15:18:07.448014 30177 tablet_bootstrap.cc:492] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e: No bootstrap required, opened a new log
I20260927 15:18:07.448312 30177 ts_tablet_manager.cc:1397] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260927 15:18:07.452010 30176 tablet_bootstrap.cc:492] T e211d836faf44c859f49edf707010309 P 17d14cf2f36c420694a271115ca2acda: Bootstrap starting.
I20260927 15:18:07.455570 30176 tablet_bootstrap.cc:654] T e211d836faf44c859f49edf707010309 P 17d14cf2f36c420694a271115ca2acda: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:07.455823 30175 raft_consensus.cc:348] T e211d836faf44c859f49edf707010309 P 6bc1d33fe45142a5a2440803335979a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc1d33fe45142a5a2440803335979a3" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36983 } } peers { permanent_uuid: "17d14cf2f36c420694a271115ca2acda" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 40231 } } peers { permanent_uuid: "6a436f04dbbc445aa2f68cd0dd81614e" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37577 } }
I20260927 15:18:07.456308 30175 raft_consensus.cc:374] T e211d836faf44c859f49edf707010309 P 6bc1d33fe45142a5a2440803335979a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:07.456467 30175 raft_consensus.cc:729] T e211d836faf44c859f49edf707010309 P 6bc1d33fe45142a5a2440803335979a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bc1d33fe45142a5a2440803335979a3, State: Initialized, Role: FOLLOWER
I20260927 15:18:07.456748 30176 log.cc:824] T e211d836faf44c859f49edf707010309 P 17d14cf2f36c420694a271115ca2acda: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:07.457163 30175 consensus_queue.cc:260] T e211d836faf44c859f49edf707010309 P 6bc1d33fe45142a5a2440803335979a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc1d33fe45142a5a2440803335979a3" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36983 } } peers { permanent_uuid: "17d14cf2f36c420694a271115ca2acda" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 40231 } } peers { permanent_uuid: "6a436f04dbbc445aa2f68cd0dd81614e" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37577 } }
I20260927 15:18:07.459230 30177 raft_consensus.cc:348] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc1d33fe45142a5a2440803335979a3" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36983 } } peers { permanent_uuid: "17d14cf2f36c420694a271115ca2acda" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 40231 } } peers { permanent_uuid: "6a436f04dbbc445aa2f68cd0dd81614e" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37577 } }
I20260927 15:18:07.459755 30177 raft_consensus.cc:374] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:07.459857 30177 raft_consensus.cc:729] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a436f04dbbc445aa2f68cd0dd81614e, State: Initialized, Role: FOLLOWER
I20260927 15:18:07.460480 30176 tablet_bootstrap.cc:492] T e211d836faf44c859f49edf707010309 P 17d14cf2f36c420694a271115ca2acda: No bootstrap required, opened a new log
I20260927 15:18:07.460759 30176 ts_tablet_manager.cc:1397] T e211d836faf44c859f49edf707010309 P 17d14cf2f36c420694a271115ca2acda: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.007s
I20260927 15:18:07.460588 30177 consensus_queue.cc:260] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc1d33fe45142a5a2440803335979a3" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36983 } } peers { permanent_uuid: "17d14cf2f36c420694a271115ca2acda" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 40231 } } peers { permanent_uuid: "6a436f04dbbc445aa2f68cd0dd81614e" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37577 } }
I20260927 15:18:07.462621 30177 ts_tablet_manager.cc:1428] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260927 15:18:07.467136 30023 heartbeater.cc:503] Master 127.10.33.126:36247 was elected leader, sending a full tablet report...
I20260927 15:18:07.469383 30175 ts_tablet_manager.cc:1428] T e211d836faf44c859f49edf707010309 P 6bc1d33fe45142a5a2440803335979a3: Time spent starting tablet: real 0.024s user 0.015s sys 0.009s
I20260927 15:18:07.470674 30176 raft_consensus.cc:348] T e211d836faf44c859f49edf707010309 P 17d14cf2f36c420694a271115ca2acda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc1d33fe45142a5a2440803335979a3" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36983 } } peers { permanent_uuid: "17d14cf2f36c420694a271115ca2acda" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 40231 } } peers { permanent_uuid: "6a436f04dbbc445aa2f68cd0dd81614e" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37577 } }
I20260927 15:18:07.471170 30176 raft_consensus.cc:374] T e211d836faf44c859f49edf707010309 P 17d14cf2f36c420694a271115ca2acda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:07.471280 30176 raft_consensus.cc:729] T e211d836faf44c859f49edf707010309 P 17d14cf2f36c420694a271115ca2acda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d14cf2f36c420694a271115ca2acda, State: Initialized, Role: FOLLOWER
I20260927 15:18:07.471923 30176 consensus_queue.cc:260] T e211d836faf44c859f49edf707010309 P 17d14cf2f36c420694a271115ca2acda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc1d33fe45142a5a2440803335979a3" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36983 } } peers { permanent_uuid: "17d14cf2f36c420694a271115ca2acda" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 40231 } } peers { permanent_uuid: "6a436f04dbbc445aa2f68cd0dd81614e" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37577 } }
I20260927 15:18:07.473091 30156 heartbeater.cc:503] Master 127.10.33.126:36247 was elected leader, sending a full tablet report...
I20260927 15:18:07.473490 30176 ts_tablet_manager.cc:1428] T e211d836faf44c859f49edf707010309 P 17d14cf2f36c420694a271115ca2acda: Time spent starting tablet: real 0.013s user 0.006s sys 0.006s
W20260927 15:18:07.540089 29890 tablet.cc:2369] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:18:07.549531 30024 tablet.cc:2369] T e211d836faf44c859f49edf707010309 P 6bc1d33fe45142a5a2440803335979a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:18:07.571496 30157 tablet.cc:2369] T e211d836faf44c859f49edf707010309 P 17d14cf2f36c420694a271115ca2acda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:18:07.577149 30182 raft_consensus.cc:482] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:18:07.577488 30182 raft_consensus.cc:504] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc1d33fe45142a5a2440803335979a3" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36983 } } peers { permanent_uuid: "17d14cf2f36c420694a271115ca2acda" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 40231 } } peers { permanent_uuid: "6a436f04dbbc445aa2f68cd0dd81614e" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37577 } }
I20260927 15:18:07.579262 30182 leader_election.cc:288] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bc1d33fe45142a5a2440803335979a3 (127.10.33.66:36983), 17d14cf2f36c420694a271115ca2acda (127.10.33.67:40231)
I20260927 15:18:07.589062 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:07.579781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:56589 (local address 127.10.33.66:36983)
0927 15:18:07.579967 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:07.579973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:07.580738 (+ 765us) server_negotiation.cc:407] Connection header received
0927 15:18:07.580921 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:07.580926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:07.581004 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:07.581105 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:07.581938 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.583053 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.586481 (+ 3428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.587758 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.587915 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:07.587993 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:07.588630 (+ 637us) server_negotiation.cc:299] Negotiation successful
0927 15:18:07.588804 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 15:18:07.589063 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:07.579781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:33799 (local address 127.10.33.67:40231)
0927 15:18:07.579967 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:07.579973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:07.580717 (+ 744us) server_negotiation.cc:407] Connection header received
0927 15:18:07.580920 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:07.580926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:07.581004 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:07.581104 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:07.581938 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.583053 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.586484 (+ 3431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.587746 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.587919 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:07.587993 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:07.588633 (+ 640us) server_negotiation.cc:299] Negotiation successful
0927 15:18:07.588791 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:18:07.589368 30184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:07.579696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:40231 (local address 127.10.33.65:33799)
0927 15:18:07.580592 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:07.580611 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:07.580808 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:07.581126 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:07.581132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:07.581178 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:07.581783 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:07.581792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.583244 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.583251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:07.586253 (+ 3002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:07.586276 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.588102 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.588110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:07.588410 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:07.588474 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 15:18:07.588587 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 15:18:07.588790 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":294,"spinlock_wait_cycles":84352,"thread_start_us":435,"threads_started":1}
I20260927 15:18:07.589389 30185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:07.579650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:36983 (local address 127.10.33.65:56589)
0927 15:18:07.580592 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:07.580611 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:07.580768 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:07.581126 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:07.581132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:07.581178 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:07.581757 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:07.581776 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.583244 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.583251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:07.586253 (+ 3002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:07.586276 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.588116 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.588122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:07.588413 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:07.588498 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 15:18:07.588590 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 15:18:07.588790 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":420,"spinlock_wait_cycles":14464,"thread_start_us":608,"threads_started":1}
I20260927 15:18:07.590773 30111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e211d836faf44c859f49edf707010309" candidate_uuid: "6a436f04dbbc445aa2f68cd0dd81614e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d14cf2f36c420694a271115ca2acda" is_pre_election: true
I20260927 15:18:07.590747 29978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e211d836faf44c859f49edf707010309" candidate_uuid: "6a436f04dbbc445aa2f68cd0dd81614e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bc1d33fe45142a5a2440803335979a3" is_pre_election: true
I20260927 15:18:07.591343 29978 raft_consensus.cc:2454] T e211d836faf44c859f49edf707010309 P 6bc1d33fe45142a5a2440803335979a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a436f04dbbc445aa2f68cd0dd81614e in term 0.
I20260927 15:18:07.591343 30111 raft_consensus.cc:2454] T e211d836faf44c859f49edf707010309 P 17d14cf2f36c420694a271115ca2acda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a436f04dbbc445aa2f68cd0dd81614e in term 0.
I20260927 15:18:07.592135 29777 leader_election.cc:302] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d14cf2f36c420694a271115ca2acda, 6a436f04dbbc445aa2f68cd0dd81614e; no voters:
I20260927 15:18:07.592592 30182 raft_consensus.cc:2781] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:18:07.592726 30182 raft_consensus.cc:482] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:18:07.592839 30182 raft_consensus.cc:3037] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:07.595101 30182 raft_consensus.cc:504] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc1d33fe45142a5a2440803335979a3" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36983 } } peers { permanent_uuid: "17d14cf2f36c420694a271115ca2acda" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 40231 } } peers { permanent_uuid: "6a436f04dbbc445aa2f68cd0dd81614e" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37577 } }
I20260927 15:18:07.595693 30182 leader_election.cc:288] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e [CANDIDATE]: Term 1 election: Requested vote from peers 6bc1d33fe45142a5a2440803335979a3 (127.10.33.66:36983), 17d14cf2f36c420694a271115ca2acda (127.10.33.67:40231)
I20260927 15:18:07.596247 30111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e211d836faf44c859f49edf707010309" candidate_uuid: "6a436f04dbbc445aa2f68cd0dd81614e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d14cf2f36c420694a271115ca2acda"
I20260927 15:18:07.596240 29978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e211d836faf44c859f49edf707010309" candidate_uuid: "6a436f04dbbc445aa2f68cd0dd81614e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bc1d33fe45142a5a2440803335979a3"
I20260927 15:18:07.596459 29978 raft_consensus.cc:3037] T e211d836faf44c859f49edf707010309 P 6bc1d33fe45142a5a2440803335979a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:07.596459 30111 raft_consensus.cc:3037] T e211d836faf44c859f49edf707010309 P 17d14cf2f36c420694a271115ca2acda [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:07.599540 29978 raft_consensus.cc:2454] T e211d836faf44c859f49edf707010309 P 6bc1d33fe45142a5a2440803335979a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a436f04dbbc445aa2f68cd0dd81614e in term 1.
I20260927 15:18:07.599539 30111 raft_consensus.cc:2454] T e211d836faf44c859f49edf707010309 P 17d14cf2f36c420694a271115ca2acda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a436f04dbbc445aa2f68cd0dd81614e in term 1.
I20260927 15:18:07.599965 29777 leader_election.cc:302] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d14cf2f36c420694a271115ca2acda, 6a436f04dbbc445aa2f68cd0dd81614e; no voters:
I20260927 15:18:07.600284 30182 raft_consensus.cc:2781] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:07.600756 30182 raft_consensus.cc:686] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e [term 1 LEADER]: Becoming Leader. State: Replica: 6a436f04dbbc445aa2f68cd0dd81614e, State: Running, Role: LEADER
I20260927 15:18:07.601253 30182 consensus_queue.cc:237] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc1d33fe45142a5a2440803335979a3" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36983 } } peers { permanent_uuid: "17d14cf2f36c420694a271115ca2acda" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 40231 } } peers { permanent_uuid: "6a436f04dbbc445aa2f68cd0dd81614e" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37577 } }
I20260927 15:18:07.605882 29699 catalog_manager.cc:5495] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a436f04dbbc445aa2f68cd0dd81614e (127.10.33.65). New cstate: current_term: 1 leader_uuid: "6a436f04dbbc445aa2f68cd0dd81614e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc1d33fe45142a5a2440803335979a3" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d14cf2f36c420694a271115ca2acda" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 40231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a436f04dbbc445aa2f68cd0dd81614e" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37577 } health_report { overall_health: HEALTHY } } }
I20260927 15:18:07.664420 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:07.658381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50216 (local address 127.10.33.65:37577)
0927 15:18:07.658586 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:07.658590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:07.658609 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:18:07.658735 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:07.658739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:07.658820 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:07.658953 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:07.659269 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.659986 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.661478 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.662075 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.662276 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:07.662882 (+ 606us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:07.663487 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:07.663623 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:07.664041 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 15:18:07.664192 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 15:18:07.673692 30111 raft_consensus.cc:1261] T e211d836faf44c859f49edf707010309 P 17d14cf2f36c420694a271115ca2acda [term 1 FOLLOWER]: Refusing update from remote peer 6a436f04dbbc445aa2f68cd0dd81614e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:18:07.673740 29978 raft_consensus.cc:1261] T e211d836faf44c859f49edf707010309 P 6bc1d33fe45142a5a2440803335979a3 [term 1 FOLLOWER]: Refusing update from remote peer 6a436f04dbbc445aa2f68cd0dd81614e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:18:07.674823 30186 consensus_queue.cc:1035] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bc1d33fe45142a5a2440803335979a3" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:18:07.675181 30182 consensus_queue.cc:1035] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d14cf2f36c420694a271115ca2acda" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 40231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:18:07.697610 30190 mvcc.cc:205] Tried to move back new op lower bound from 7333979290298540032 to 7333979290018496512. Current Snapshot: MvccSnapshot[applied={T|T < 7333979290298540032 or (T in {7333979290298540032})}]
I20260927 15:18:07.706540 29699 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:18:07.706892 29699 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:18:07.711769 29699 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7de2e3ef43de4d4287b876b61ddfb19b: Sending DeleteTablet for 3 replicas of tablet e211d836faf44c859f49edf707010309
I20260927 15:18:07.713037 30091 tablet_service.cc:1514] Processing DeleteTablet for tablet e211d836faf44c859f49edf707010309 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:18:07 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:57360
I20260927 15:18:07.713207 29824 tablet_service.cc:1514] Processing DeleteTablet for tablet e211d836faf44c859f49edf707010309 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:18:07 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:50206
I20260927 15:18:07.713999 30198 tablet_replica.cc:330] T e211d836faf44c859f49edf707010309 P 17d14cf2f36c420694a271115ca2acda: stopping tablet replica
I20260927 15:18:07.713970 29958 tablet_service.cc:1514] Processing DeleteTablet for tablet e211d836faf44c859f49edf707010309 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:18:07 UTC) from {username='kudu', principal='kudu/127.10.33.126@KRBTEST.COM'} at 127.0.0.1:37514
I20260927 15:18:07.714490 30198 raft_consensus.cc:2229] T e211d836faf44c859f49edf707010309 P 17d14cf2f36c420694a271115ca2acda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:18:07.715090 30198 raft_consensus.cc:2258] T e211d836faf44c859f49edf707010309 P 17d14cf2f36c420694a271115ca2acda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:18:07.715770 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 29761
I20260927 15:18:07.719094 30199 tablet_replica.cc:330] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e: stopping tablet replica
I20260927 15:18:07.719673 30199 raft_consensus.cc:2229] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:18:07.720607 30199 raft_consensus.cc:2258] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:18:07.723142 30199 ts_tablet_manager.cc:1905] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:18:07.723501 30198 ts_tablet_manager.cc:1905] T e211d836faf44c859f49edf707010309 P 17d14cf2f36c420694a271115ca2acda: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 15:18:07 dist-test-slave-f5h6 krb5kdc[29655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.65@KRBTEST.COM
I20260927 15:18:07.732641 30199 ts_tablet_manager.cc:1918] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:18:07.733176 30199 log.cc:1196] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/e211d836faf44c859f49edf707010309
I20260927 15:18:07.733777 30199 ts_tablet_manager.cc:1939] T e211d836faf44c859f49edf707010309 P 6a436f04dbbc445aa2f68cd0dd81614e: Deleting consensus metadata
I20260927 15:18:07.734195 30200 tablet_replica.cc:330] T e211d836faf44c859f49edf707010309 P 6bc1d33fe45142a5a2440803335979a3: stopping tablet replica
I20260927 15:18:07.734781 30200 raft_consensus.cc:2229] T e211d836faf44c859f49edf707010309 P 6bc1d33fe45142a5a2440803335979a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:18:07.735558 30200 raft_consensus.cc:2258] T e211d836faf44c859f49edf707010309 P 6bc1d33fe45142a5a2440803335979a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:18:07.735958 29687 catalog_manager.cc:4841] TS 6a436f04dbbc445aa2f68cd0dd81614e (127.10.33.65:37577): tablet e211d836faf44c859f49edf707010309 (table test-table [id=76ad30a2df0f47e0a60e0cc1ace3d8e9]) successfully deleted
I20260927 15:18:07.739403 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:07.716790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50218 (local address 127.10.33.65:37577)
0927 15:18:07.717206 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:07.717214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:07.718446 (+ 1232us) server_negotiation.cc:407] Connection header received
0927 15:18:07.718610 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:07.718616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:07.718726 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:07.718858 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:07.720566 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.721772 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.723624 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.724330 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.724435 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:07.730752 (+ 6317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:07.730773 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:07.730793 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:07.730853 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:07.735015 (+ 4162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:07.736117 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:07.736126 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:07.736130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:07.736393 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:07.736895 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:07.736905 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:07.736909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:07.737316 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:07.737723 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:07.738674 (+ 951us) server_negotiation.cc:299] Negotiation successful
0927 15:18:07.738936 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248}
I20260927 15:18:07.739382 30198 ts_tablet_manager.cc:1918] T e211d836faf44c859f49edf707010309 P 17d14cf2f36c420694a271115ca2acda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:18:07.739583 30198 log.cc:1196] T e211d836faf44c859f49edf707010309 P 17d14cf2f36c420694a271115ca2acda: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/e211d836faf44c859f49edf707010309
I20260927 15:18:07.740142 29884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:07.742194 30200 ts_tablet_manager.cc:1905] T e211d836faf44c859f49edf707010309 P 6bc1d33fe45142a5a2440803335979a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:18:07.749253 30200 ts_tablet_manager.cc:1918] T e211d836faf44c859f49edf707010309 P 6bc1d33fe45142a5a2440803335979a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:18:07.749451 30200 log.cc:1196] T e211d836faf44c859f49edf707010309 P 6bc1d33fe45142a5a2440803335979a3: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/e211d836faf44c859f49edf707010309
I20260927 15:18:07.749953 30200 ts_tablet_manager.cc:1939] T e211d836faf44c859f49edf707010309 P 6bc1d33fe45142a5a2440803335979a3: Deleting consensus metadata
I20260927 15:18:07.751873 29686 catalog_manager.cc:4841] TS 6bc1d33fe45142a5a2440803335979a3 (127.10.33.66:36983): tablet e211d836faf44c859f49edf707010309 (table test-table [id=76ad30a2df0f47e0a60e0cc1ace3d8e9]) successfully deleted
I20260927 15:18:07.752189 30198 ts_tablet_manager.cc:1939] T e211d836faf44c859f49edf707010309 P 17d14cf2f36c420694a271115ca2acda: Deleting consensus metadata
I20260927 15:18:07.754166 29684 catalog_manager.cc:4841] TS 17d14cf2f36c420694a271115ca2acda (127.10.33.67:40231): tablet e211d836faf44c859f49edf707010309 (table test-table [id=76ad30a2df0f47e0a60e0cc1ace3d8e9]) successfully deleted
I20260927 15:18:07.855119 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 29761
I20260927 15:18:07.871366 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 29895
Sep 27 15:18:07 dist-test-slave-f5h6 krb5kdc[29655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.66@KRBTEST.COM
I20260927 15:18:07.886953 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:07.872266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37526 (local address 127.10.33.66:36983)
0927 15:18:07.872431 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:07.872435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:07.872453 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:18:07.872516 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:07.872520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:07.872597 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:07.872685 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:07.873978 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.874786 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.876121 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:07.876563 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:07.876673 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:07.881141 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:07.881159 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:07.881175 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:07.881220 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:07.884196 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:07.884957 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:07.884965 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:07.884968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:07.885125 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:07.885470 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:07.885473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:07.885475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:07.885929 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:07.886153 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:07.886467 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 15:18:07.886649 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:18:07.887732 30018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:07.978482 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 29895
I20260927 15:18:07.996970 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 30028
Sep 27 15:18:08 dist-test-slave-f5h6 krb5kdc[29655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.67@KRBTEST.COM
I20260927 15:18:08.014017 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:07.998037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.10.33.67:40231)
0927 15:18:07.998276 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:07.998282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:07.998306 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:18:07.998379 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:07.998385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:07.998469 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:07.998558 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:08.000428 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:08.001405 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:08.002901 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:08.003309 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:08.003409 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:08.008030 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:08.008046 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:08.008064 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:08.008110 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:08.011043 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:08.011949 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:08.011959 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:08.011962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:08.012123 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:08.012491 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:08.012494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:08.012497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:08.012877 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:08.013099 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:08.013390 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 15:18:08.013568 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 15:18:08.014631 30151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:08.105826 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 30028
I20260927 15:18:08.120172 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 29668
I20260927 15:18:08.121310 29730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:08.232558 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 29668
2026-09-27T15:18:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4269 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T15:18:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:08Z Disabled control of system clock
I20260927 15:18:08.304486 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:39417
--webserver_interface=127.10.33.126
--webserver_port=0
--builtin_ntp_servers=127.10.33.84:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:39417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:18:08.640021 30209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:08.640362 30209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:08.640446 30209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:08.647928 30209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:08.648047 30209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:08.648084 30209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:08.648132 30209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:08.648168 30209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:08.657764 30209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:39417
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:39417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:08.659142 30209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:08.660918 30209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:08.672801 30214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:08.672816 30215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:08.673520 30217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:08.674045 30209 server_base.cc:1034] running on GCE node
I20260927 15:18:08.674852 30209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:08.676471 30209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:08.677734 30209 hybrid_clock.cc:648] HybridClock initialized: now 1790522288677680 us; error 48 us; skew 500 ppm
I20260927 15:18:08.678154 30209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:18:08.680871 30209 webserver.cc:460] Webserver started at http://127.10.33.126:46151/ using document root <none> and password file <none>
I20260927 15:18:08.681641 30209 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:08.681749 30209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:08.682104 30209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:08.685034 30209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "0da698a8c79f41d885b9b337e28c1ff9"
format_stamp: "Formatted at 2026-09-27 15:18:08 on dist-test-slave-f5h6"
server_key: "efee863928616ad2320467bae17a4c86"
server_key_iv: "9b241ca609d68ba026e46a8d7624dff3"
server_key_version: "encryptionkey@0"
I20260927 15:18:08.685849 30209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "0da698a8c79f41d885b9b337e28c1ff9"
format_stamp: "Formatted at 2026-09-27 15:18:08 on dist-test-slave-f5h6"
server_key: "efee863928616ad2320467bae17a4c86"
server_key_iv: "9b241ca609d68ba026e46a8d7624dff3"
server_key_version: "encryptionkey@0"
I20260927 15:18:08.691118 30209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:18:08.694825 30223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:08.696305 30209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:18:08.696483 30209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "0da698a8c79f41d885b9b337e28c1ff9"
format_stamp: "Formatted at 2026-09-27 15:18:08 on dist-test-slave-f5h6"
server_key: "efee863928616ad2320467bae17a4c86"
server_key_iv: "9b241ca609d68ba026e46a8d7624dff3"
server_key_version: "encryptionkey@0"
I20260927 15:18:08.696699 30209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:08.718920 30209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:08.719835 30209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:08.720129 30209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:08.738592 30209 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:39417
I20260927 15:18:08.738628 30274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:39417 every 8 connection(s)
I20260927 15:18:08.739998 30209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:08.744072 30275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:08.749789 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 30209
I20260927 15:18:08.750010 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:08.750468 10373 external_mini_cluster.cc:1428] Setting key c5c4ac13024b40f8182e4d90cb5066ac
I20260927 15:18:08.757213 30275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9: Bootstrap starting.
I20260927 15:18:08.761541 30275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:08.762871 30275 log.cc:824] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:08.767427 30275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9: No bootstrap required, opened a new log
I20260927 15:18:08.768496 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:08.752634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.10.33.126:39417)
0927 15:18:08.753919 (+ 1285us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:08.753936 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:08.754000 (+ 64us) server_negotiation.cc:407] Connection header received
0927 15:18:08.756457 (+ 2457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:08.756495 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:08.757169 (+ 674us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:08.757635 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:08.758259 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:08.760549 (+ 2290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:08.762116 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:08.763182 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:08.763385 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:08.764784 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:08.764795 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:08.764816 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:08.765547 (+ 731us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:08.765681 (+ 134us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:08.765713 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:08.765846 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:08.766426 (+ 580us) server_negotiation.cc:299] Negotiation successful
0927 15:18:08.766961 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":413,"threads_started":1}
I20260927 15:18:08.776537 30275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da698a8c79f41d885b9b337e28c1ff9" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39417 } }
I20260927 15:18:08.776947 30275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:08.777029 30275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da698a8c79f41d885b9b337e28c1ff9, State: Initialized, Role: FOLLOWER
I20260927 15:18:08.777606 30275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da698a8c79f41d885b9b337e28c1ff9" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39417 } }
I20260927 15:18:08.777849 30275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:18:08.778040 30275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:18:08.778255 30275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:08.780894 30275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da698a8c79f41d885b9b337e28c1ff9" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39417 } }
I20260927 15:18:08.781466 30275 leader_election.cc:302] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0da698a8c79f41d885b9b337e28c1ff9; no voters:
I20260927 15:18:08.782104 30275 leader_election.cc:288] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:18:08.782254 30280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:08.783344 30280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 0da698a8c79f41d885b9b337e28c1ff9, State: Running, Role: LEADER
I20260927 15:18:08.783941 30280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da698a8c79f41d885b9b337e28c1ff9" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39417 } }
I20260927 15:18:08.784377 30275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:08.786535 30281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0da698a8c79f41d885b9b337e28c1ff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da698a8c79f41d885b9b337e28c1ff9" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39417 } } }
I20260927 15:18:08.787698 30281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:08.786994 30282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0da698a8c79f41d885b9b337e28c1ff9. Latest consensus state: current_term: 1 leader_uuid: "0da698a8c79f41d885b9b337e28c1ff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da698a8c79f41d885b9b337e28c1ff9" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 39417 } } }
I20260927 15:18:08.788188 30282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:08.790146 30289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:08.795707 30289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:08.804757 30289 catalog_manager.cc:1347] Generated new cluster ID: 1f8bae8b027c4559a392de2ac88b849d
I20260927 15:18:08.804903 30289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:08.832556 30289 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:08.833911 30289 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:08.843276 30289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9: Generated new TSK 0
I20260927 15:18:08.844223 30289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:18:08.857625 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.65:0
--local_ip_for_outbound_sockets=127.10.33.65
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:39417
--builtin_ntp_servers=127.10.33.84:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:18:09.221927 30299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:09.222226 30299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:09.222343 30299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:09.229682 30299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:09.229795 30299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:09.229952 30299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.65
I20260927 15:18:09.239902 30299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.33.65
--webserver_port=0
--tserver_master_addrs=127.10.33.126:39417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.65
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:09.241343 30299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:09.243323 30299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:09.257522 30304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:09.257484 30305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:09.257550 30307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:09.258260 30299 server_base.cc:1034] running on GCE node
I20260927 15:18:09.258915 30299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:09.259904 30299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:09.261144 30299 hybrid_clock.cc:648] HybridClock initialized: now 1790522289261105 us; error 27 us; skew 500 ppm
I20260927 15:18:09.261516 30299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:18:09.263926 30299 webserver.cc:460] Webserver started at http://127.10.33.65:35729/ using document root <none> and password file <none>
I20260927 15:18:09.264627 30299 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:09.264729 30299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:09.265024 30299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:09.268105 30299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-0/data/instance:
uuid: "bcba7828df294639bdb0a503a03ac852"
format_stamp: "Formatted at 2026-09-27 15:18:09 on dist-test-slave-f5h6"
server_key: "737e4594e04607481e6e07d0af54cc50"
server_key_iv: "b4b8aac6ab75e6e5d6ce46f7808617fe"
server_key_version: "encryptionkey@0"
I20260927 15:18:09.268910 30299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance:
uuid: "bcba7828df294639bdb0a503a03ac852"
format_stamp: "Formatted at 2026-09-27 15:18:09 on dist-test-slave-f5h6"
server_key: "737e4594e04607481e6e07d0af54cc50"
server_key_iv: "b4b8aac6ab75e6e5d6ce46f7808617fe"
server_key_version: "encryptionkey@0"
I20260927 15:18:09.274187 30299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 15:18:09.277503 30313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:09.278982 30299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:18:09.279160 30299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-0/wal
uuid: "bcba7828df294639bdb0a503a03ac852"
format_stamp: "Formatted at 2026-09-27 15:18:09 on dist-test-slave-f5h6"
server_key: "737e4594e04607481e6e07d0af54cc50"
server_key_iv: "b4b8aac6ab75e6e5d6ce46f7808617fe"
server_key_version: "encryptionkey@0"
I20260927 15:18:09.279382 30299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:09.301450 30299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:09.302341 30299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:09.302623 30299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:09.303596 30299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:18:09.305064 30299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:18:09.305159 30299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:09.305308 30299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:18:09.305378 30299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:09.335144 30299 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.65:37137
I20260927 15:18:09.335228 30425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.65:37137 every 8 connection(s)
I20260927 15:18:09.336519 30299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-0/data/info.pb
I20260927 15:18:09.346601 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 30299
I20260927 15:18:09.346782 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-0/wal/instance
I20260927 15:18:09.347199 10373 external_mini_cluster.cc:1428] Setting key 59546fbeca6c2d6234442dfa857ee67a
I20260927 15:18:09.349123 30428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:09.339211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:40801 (local address 127.10.33.126:39417)
0927 15:18:09.339666 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:09.339671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:09.340720 (+ 1049us) server_negotiation.cc:407] Connection header received
0927 15:18:09.342525 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:09.342555 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:09.342648 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:09.342770 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:09.343792 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:09.344491 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:09.346069 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:09.346650 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:09.346776 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:09.348192 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:09.348198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:09.348202 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:09.348258 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:09.348309 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:09.348327 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:09.348424 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:09.348667 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 15:18:09.348774 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":207,"threads_started":1}
I20260927 15:18:09.350236 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.66:0
--local_ip_for_outbound_sockets=127.10.33.66
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:39417
--builtin_ntp_servers=127.10.33.84:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:18:09.350440 30429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:09.339610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39417 (local address 127.10.33.65:40801)
0927 15:18:09.340521 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:09.340574 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:09.342261 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:09.342985 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:09.342994 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:09.343082 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:09.343631 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:09.343649 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:09.344639 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:09.344643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:09.345927 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:09.345940 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:09.346912 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:09.346919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:09.347124 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:09.347829 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:18:09.347832 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:09.347874 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:18:09.347980 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:09.348458 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:09.348468 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:09.348475 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:18:09.348636 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 15:18:09.349038 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":272,"threads_started":1}
I20260927 15:18:09.351881 30426 heartbeater.cc:347] Connected to a master server at 127.10.33.126:39417
I20260927 15:18:09.352157 30426 heartbeater.cc:464] Registering TS with master...
I20260927 15:18:09.352862 30426 heartbeater.cc:511] Master 127.10.33.126:39417 requested a full tablet report, sending...
I20260927 15:18:09.354857 30240 ts_manager.cc:194] Registered new tserver with Master: bcba7828df294639bdb0a503a03ac852 (127.10.33.65:37137)
I20260927 15:18:09.356292 30240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.33.65:40801
W20260927 15:18:09.670933 30430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:09.671226 30430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:09.671334 30430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:09.678648 30430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:09.678755 30430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:09.678861 30430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.66
I20260927 15:18:09.688154 30430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.33.66
--webserver_port=0
--tserver_master_addrs=127.10.33.126:39417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.66
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:09.689419 30430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:09.691335 30430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:09.704126 30436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:09.704231 30435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:09.704375 30438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:09.704844 30430 server_base.cc:1034] running on GCE node
I20260927 15:18:09.705412 30430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:09.706373 30430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:09.707609 30430 hybrid_clock.cc:648] HybridClock initialized: now 1790522289707550 us; error 33 us; skew 500 ppm
I20260927 15:18:09.707906 30430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:18:09.710307 30430 webserver.cc:460] Webserver started at http://127.10.33.66:34823/ using document root <none> and password file <none>
I20260927 15:18:09.710948 30430 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:09.711043 30430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:09.711301 30430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:09.714222 30430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-1/data/instance:
uuid: "534b336b1dd041b5a37e593e23b06c7f"
format_stamp: "Formatted at 2026-09-27 15:18:09 on dist-test-slave-f5h6"
server_key: "1141e92bda9b9540297e14b27c9e4ed7"
server_key_iv: "009b73a8fd26bfe150579fa84e853372"
server_key_version: "encryptionkey@0"
I20260927 15:18:09.715061 30430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance:
uuid: "534b336b1dd041b5a37e593e23b06c7f"
format_stamp: "Formatted at 2026-09-27 15:18:09 on dist-test-slave-f5h6"
server_key: "1141e92bda9b9540297e14b27c9e4ed7"
server_key_iv: "009b73a8fd26bfe150579fa84e853372"
server_key_version: "encryptionkey@0"
I20260927 15:18:09.720466 30430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:18:09.724186 30444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:09.725620 30430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:18:09.725760 30430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-1/wal
uuid: "534b336b1dd041b5a37e593e23b06c7f"
format_stamp: "Formatted at 2026-09-27 15:18:09 on dist-test-slave-f5h6"
server_key: "1141e92bda9b9540297e14b27c9e4ed7"
server_key_iv: "009b73a8fd26bfe150579fa84e853372"
server_key_version: "encryptionkey@0"
I20260927 15:18:09.726027 30430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:09.743592 30430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:09.744436 30430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:09.744716 30430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:09.745723 30430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:18:09.747148 30430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:18:09.747232 30430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:09.747331 30430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:18:09.747394 30430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:09.777705 30430 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.66:38385
I20260927 15:18:09.777768 30556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.66:38385 every 8 connection(s)
I20260927 15:18:09.779000 30430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-1/data/info.pb
I20260927 15:18:09.785845 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 30430
I20260927 15:18:09.786044 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-1/wal/instance
I20260927 15:18:09.786438 10373 external_mini_cluster.cc:1428] Setting key 3b6bc301f0b1bf6a03543e9856b464fd
I20260927 15:18:09.789404 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.33.67:0
--local_ip_for_outbound_sockets=127.10.33.67
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:39417
--builtin_ntp_servers=127.10.33.84:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:18:09.792125 30428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:09.781646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.66:45517 (local address 127.10.33.126:39417)
0927 15:18:09.781829 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:09.781834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:09.783187 (+ 1353us) server_negotiation.cc:407] Connection header received
0927 15:18:09.784978 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:09.784983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:09.785054 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:09.785177 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:09.786345 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:09.787114 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:09.789154 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:09.789766 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:09.789914 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:09.791356 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:09.791363 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:09.791367 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:09.791389 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:09.791439 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:09.791455 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:09.791543 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:09.791755 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 15:18:09.791849 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 15:18:09.793061 30559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:09.782112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39417 (local address 127.10.33.66:45517)
0927 15:18:09.783036 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:09.783081 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:09.784735 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:09.785445 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:09.785454 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:09.785534 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:09.786186 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:09.786206 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:09.787266 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:09.787270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:09.788965 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:09.788977 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:09.790023 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:09.790032 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:09.790297 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:09.790986 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:18:09.790989 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:09.791036 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:18:09.791164 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:09.791555 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:09.791565 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:09.791571 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:18:09.791721 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 15:18:09.792017 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":267,"threads_started":1}
I20260927 15:18:09.794530 30557 heartbeater.cc:347] Connected to a master server at 127.10.33.126:39417
I20260927 15:18:09.794847 30557 heartbeater.cc:464] Registering TS with master...
I20260927 15:18:09.795589 30557 heartbeater.cc:511] Master 127.10.33.126:39417 requested a full tablet report, sending...
I20260927 15:18:09.796926 30240 ts_manager.cc:194] Registered new tserver with Master: 534b336b1dd041b5a37e593e23b06c7f (127.10.33.66:38385)
I20260927 15:18:09.797696 30240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.33.66:45517
W20260927 15:18:10.119827 30560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:10.120151 30560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:10.120277 30560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:10.127729 30560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:10.127857 30560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:10.127993 30560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.33.67
I20260927 15:18:10.137435 30560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.33.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.33.67
--webserver_port=0
--tserver_master_addrs=127.10.33.126:39417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.33.67
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:10.138749 30560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:10.140486 30560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:10.153292 30565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:10.153384 30568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:10.153403 30566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:10.154528 30560 server_base.cc:1034] running on GCE node
I20260927 15:18:10.155121 30560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:10.156072 30560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:10.157291 30560 hybrid_clock.cc:648] HybridClock initialized: now 1790522290157247 us; error 39 us; skew 500 ppm
I20260927 15:18:10.157617 30560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 15:18:10.159951 30560 webserver.cc:460] Webserver started at http://127.10.33.67:43543/ using document root <none> and password file <none>
I20260927 15:18:10.160627 30560 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:10.160718 30560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:10.161000 30560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:10.163938 30560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-2/data/instance:
uuid: "104a07fb9b3141c98e675e0b15390fc1"
format_stamp: "Formatted at 2026-09-27 15:18:10 on dist-test-slave-f5h6"
server_key: "990c9add7492f3cdc77f4b5fc8b60b89"
server_key_iv: "cc8bd077bf031d723672657eba7fcc1f"
server_key_version: "encryptionkey@0"
I20260927 15:18:10.164750 30560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance:
uuid: "104a07fb9b3141c98e675e0b15390fc1"
format_stamp: "Formatted at 2026-09-27 15:18:10 on dist-test-slave-f5h6"
server_key: "990c9add7492f3cdc77f4b5fc8b60b89"
server_key_iv: "cc8bd077bf031d723672657eba7fcc1f"
server_key_version: "encryptionkey@0"
I20260927 15:18:10.170070 30560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260927 15:18:10.173913 30574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:10.175382 30560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:18:10.175554 30560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-2/wal
uuid: "104a07fb9b3141c98e675e0b15390fc1"
format_stamp: "Formatted at 2026-09-27 15:18:10 on dist-test-slave-f5h6"
server_key: "990c9add7492f3cdc77f4b5fc8b60b89"
server_key_iv: "cc8bd077bf031d723672657eba7fcc1f"
server_key_version: "encryptionkey@0"
I20260927 15:18:10.175804 30560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:10.214154 30560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:10.215101 30560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:10.215402 30560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:10.216488 30560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:18:10.218046 30560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:18:10.218144 30560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:10.218255 30560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:18:10.218333 30560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:10.250216 30560 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.67:34933
I20260927 15:18:10.250252 30686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.67:34933 every 8 connection(s)
I20260927 15:18:10.251526 30560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-2/data/info.pb
I20260927 15:18:10.255824 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 30560
I20260927 15:18:10.256019 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-2/wal/instance
I20260927 15:18:10.256590 10373 external_mini_cluster.cc:1428] Setting key b326b0f75eb8d9e7ed556175e29c21a3
I20260927 15:18:10.264798 30428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.254210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:42795 (local address 127.10.33.126:39417)
0927 15:18:10.254426 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:10.254431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:10.256039 (+ 1608us) server_negotiation.cc:407] Connection header received
0927 15:18:10.258355 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:10.258361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:10.258439 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:10.258524 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:10.259708 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.260456 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.262081 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.262489 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.262606 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:10.263952 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:10.263957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:10.263960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:10.263979 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:10.264016 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:10.264028 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:10.264109 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:10.264412 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 15:18:10.264517 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 15:18:10.265786 30689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.254747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:39417 (local address 127.10.33.67:42795)
0927 15:18:10.255851 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:10.255899 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:10.258042 (+ 2143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:10.258762 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:10.258771 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:10.258864 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:10.259552 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:10.259576 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.260618 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.260642 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:10.261922 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:10.261938 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.262718 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.262723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:10.262941 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:10.263611 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:18:10.263615 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:10.263656 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:18:10.263745 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:10.264147 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:10.264158 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:10.264164 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:18:10.264344 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 15:18:10.264695 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":299,"threads_started":1}
I20260927 15:18:10.267308 30687 heartbeater.cc:347] Connected to a master server at 127.10.33.126:39417
I20260927 15:18:10.267639 30687 heartbeater.cc:464] Registering TS with master...
I20260927 15:18:10.268433 30687 heartbeater.cc:511] Master 127.10.33.126:39417 requested a full tablet report, sending...
I20260927 15:18:10.270134 30240 ts_manager.cc:194] Registered new tserver with Master: 104a07fb9b3141c98e675e0b15390fc1 (127.10.33.67:34933)
I20260927 15:18:10.271240 30240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.33.67:42795
I20260927 15:18:10.272001 10373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:18:10.282893 30428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.276730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56226 (local address 127.10.33.126:39417)
0927 15:18:10.276959 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:10.276963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:10.277396 (+ 433us) server_negotiation.cc:407] Connection header received
0927 15:18:10.277507 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:10.277511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:10.277599 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:10.277710 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:10.278291 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.279084 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.280530 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.280996 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.281113 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:10.282169 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:10.282176 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:10.282181 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:10.282215 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:10.282259 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:10.282274 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:10.282352 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:10.282481 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 15:18:10.282582 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:18:10.287113 30239 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:18:10.294690 30239 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:18:10.334357 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.309076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33964 (local address 127.10.33.66:38385)
0927 15:18:10.326853 (+ 17777us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:10.326862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:10.326893 (+ 31us) server_negotiation.cc:407] Connection header received
0927 15:18:10.326993 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:10.327001 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:10.327252 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:10.327454 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:10.328107 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.329323 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.332032 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.333146 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.333267 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:10.333530 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:10.333814 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 15:18:10.334025 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17623,"thread_start_us":17518,"threads_started":1}
I20260927 15:18:10.334860 30700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.307939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38420 (local address 127.10.33.67:34933)
0927 15:18:10.325297 (+ 17358us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:10.325308 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:10.325348 (+ 40us) server_negotiation.cc:407] Connection header received
0927 15:18:10.325468 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:10.325478 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:10.325782 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:10.327067 (+ 1285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:10.327334 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.328854 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.331969 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.333264 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.333403 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:10.333685 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:10.334380 (+ 695us) server_negotiation.cc:299] Negotiation successful
0927 15:18:10.334560 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17162,"thread_start_us":17018,"threads_started":1}
I20260927 15:18:10.335026 30701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.307972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:37137 (local address 127.0.0.1:37600)
0927 15:18:10.319413 (+ 11441us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:10.319431 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:10.319557 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:10.327858 (+ 8301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:10.327862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:10.327887 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:10.328242 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:10.328250 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.330022 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.330027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:10.332283 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:10.332296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.334154 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.334161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:10.334326 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:10.334360 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 15:18:10.334426 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 15:18:10.334555 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11318,"mutex_wait_us":3,"thread_start_us":11192,"threads_started":1}
I20260927 15:18:10.335079 30702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.307707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:34933 (local address 127.0.0.1:38420)
0927 15:18:10.318512 (+ 10805us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:10.318583 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:10.318840 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:10.326290 (+ 7450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:10.326300 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:10.326337 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:10.326849 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:10.326863 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.329033 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.329039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:10.331761 (+ 2722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:10.331777 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.333678 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.333685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:10.333846 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:10.333902 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 15:18:10.333974 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 15:18:10.334088 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10418,"mutex_wait_us":99,"thread_start_us":18272,"threads_started":1}
I20260927 15:18:10.335631 30699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.308413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:38385 (local address 127.0.0.1:33964)
0927 15:18:10.324551 (+ 16138us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:10.324570 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:10.324692 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:10.327451 (+ 2759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:10.327456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:10.327481 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:10.327894 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:10.327906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.329467 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.329471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:10.331793 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:10.331809 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.333329 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.333334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:10.333542 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:10.333598 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 15:18:10.333731 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 15:18:10.333962 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15995,"mutex_wait_us":5,"thread_start_us":9542,"threads_started":1}
I20260927 15:18:10.336041 30703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.308557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37600 (local address 127.10.33.65:37137)
0927 15:18:10.327291 (+ 18734us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:10.327300 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:10.327325 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:18:10.327416 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:10.327423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:10.327624 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:10.328379 (+ 755us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:10.328544 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.329850 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.332475 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.333928 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.334805 (+ 877us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:10.335020 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:10.335097 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 15:18:10.335266 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18517,"thread_start_us":19053,"threads_started":1}
I20260927 15:18:10.339949 30492 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5d45e8953d426aa2de78a7461d64da (DEFAULT_TABLE table=test-table [id=d6422b4b1c5640b4925867383edb1fff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:18:10.339947 30622 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5d45e8953d426aa2de78a7461d64da (DEFAULT_TABLE table=test-table [id=d6422b4b1c5640b4925867383edb1fff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:18:10.340184 30361 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5d45e8953d426aa2de78a7461d64da (DEFAULT_TABLE table=test-table [id=d6422b4b1c5640b4925867383edb1fff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:18:10.341514 30492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5d45e8953d426aa2de78a7461d64da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:10.341514 30361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5d45e8953d426aa2de78a7461d64da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:10.342155 30622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5d45e8953d426aa2de78a7461d64da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:10.355984 30705 tablet_bootstrap.cc:492] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852: Bootstrap starting.
I20260927 15:18:10.357509 30707 tablet_bootstrap.cc:492] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1: Bootstrap starting.
I20260927 15:18:10.359203 30706 tablet_bootstrap.cc:492] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f: Bootstrap starting.
I20260927 15:18:10.360014 30426 heartbeater.cc:503] Master 127.10.33.126:39417 was elected leader, sending a full tablet report...
I20260927 15:18:10.361809 30707 tablet_bootstrap.cc:654] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:10.362835 30706 tablet_bootstrap.cc:654] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:10.362861 30705 tablet_bootstrap.cc:654] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:10.363221 30707 log.cc:824] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:10.364084 30706 log.cc:824] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:10.364077 30705 log.cc:824] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:10.368494 30706 tablet_bootstrap.cc:492] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f: No bootstrap required, opened a new log
I20260927 15:18:10.368508 30705 tablet_bootstrap.cc:492] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852: No bootstrap required, opened a new log
I20260927 15:18:10.368803 30705 ts_tablet_manager.cc:1397] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20260927 15:18:10.368803 30707 tablet_bootstrap.cc:492] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1: No bootstrap required, opened a new log
I20260927 15:18:10.368826 30706 ts_tablet_manager.cc:1397] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 15:18:10.369052 30707 ts_tablet_manager.cc:1397] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260927 15:18:10.376434 30705 raft_consensus.cc:348] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104a07fb9b3141c98e675e0b15390fc1" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34933 } } peers { permanent_uuid: "bcba7828df294639bdb0a503a03ac852" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37137 } } peers { permanent_uuid: "534b336b1dd041b5a37e593e23b06c7f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 38385 } }
I20260927 15:18:10.376848 30705 raft_consensus.cc:374] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:10.376971 30705 raft_consensus.cc:729] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcba7828df294639bdb0a503a03ac852, State: Initialized, Role: FOLLOWER
I20260927 15:18:10.377666 30705 consensus_queue.cc:260] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104a07fb9b3141c98e675e0b15390fc1" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34933 } } peers { permanent_uuid: "bcba7828df294639bdb0a503a03ac852" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37137 } } peers { permanent_uuid: "534b336b1dd041b5a37e593e23b06c7f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 38385 } }
I20260927 15:18:10.379830 30706 raft_consensus.cc:348] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104a07fb9b3141c98e675e0b15390fc1" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34933 } } peers { permanent_uuid: "bcba7828df294639bdb0a503a03ac852" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37137 } } peers { permanent_uuid: "534b336b1dd041b5a37e593e23b06c7f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 38385 } }
I20260927 15:18:10.379830 30707 raft_consensus.cc:348] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104a07fb9b3141c98e675e0b15390fc1" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34933 } } peers { permanent_uuid: "bcba7828df294639bdb0a503a03ac852" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37137 } } peers { permanent_uuid: "534b336b1dd041b5a37e593e23b06c7f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 38385 } }
I20260927 15:18:10.380333 30707 raft_consensus.cc:374] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:10.380333 30706 raft_consensus.cc:374] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:10.380438 30706 raft_consensus.cc:729] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 534b336b1dd041b5a37e593e23b06c7f, State: Initialized, Role: FOLLOWER
I20260927 15:18:10.380438 30707 raft_consensus.cc:729] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 104a07fb9b3141c98e675e0b15390fc1, State: Initialized, Role: FOLLOWER
I20260927 15:18:10.381156 30706 consensus_queue.cc:260] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104a07fb9b3141c98e675e0b15390fc1" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34933 } } peers { permanent_uuid: "bcba7828df294639bdb0a503a03ac852" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37137 } } peers { permanent_uuid: "534b336b1dd041b5a37e593e23b06c7f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 38385 } }
I20260927 15:18:10.381222 30707 consensus_queue.cc:260] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104a07fb9b3141c98e675e0b15390fc1" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34933 } } peers { permanent_uuid: "bcba7828df294639bdb0a503a03ac852" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37137 } } peers { permanent_uuid: "534b336b1dd041b5a37e593e23b06c7f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 38385 } }
I20260927 15:18:10.382479 30557 heartbeater.cc:503] Master 127.10.33.126:39417 was elected leader, sending a full tablet report...
I20260927 15:18:10.383092 30687 heartbeater.cc:503] Master 127.10.33.126:39417 was elected leader, sending a full tablet report...
I20260927 15:18:10.383232 30706 ts_tablet_manager.cc:1428] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f: Time spent starting tablet: real 0.014s user 0.010s sys 0.006s
I20260927 15:18:10.383567 30707 ts_tablet_manager.cc:1428] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260927 15:18:10.387382 30705 ts_tablet_manager.cc:1428] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852: Time spent starting tablet: real 0.018s user 0.009s sys 0.008s
I20260927 15:18:10.468508 30711 raft_consensus.cc:482] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:18:10.468812 30711 raft_consensus.cc:504] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104a07fb9b3141c98e675e0b15390fc1" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34933 } } peers { permanent_uuid: "bcba7828df294639bdb0a503a03ac852" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37137 } } peers { permanent_uuid: "534b336b1dd041b5a37e593e23b06c7f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 38385 } }
I20260927 15:18:10.470819 30711 leader_election.cc:288] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 104a07fb9b3141c98e675e0b15390fc1 (127.10.33.67:34933), 534b336b1dd041b5a37e593e23b06c7f (127.10.33.66:38385)
I20260927 15:18:10.475890 30713 raft_consensus.cc:482] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:18:10.476253 30713 raft_consensus.cc:504] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104a07fb9b3141c98e675e0b15390fc1" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34933 } } peers { permanent_uuid: "bcba7828df294639bdb0a503a03ac852" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37137 } } peers { permanent_uuid: "534b336b1dd041b5a37e593e23b06c7f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 38385 } }
I20260927 15:18:10.478307 30713 leader_election.cc:288] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcba7828df294639bdb0a503a03ac852 (127.10.33.65:37137), 534b336b1dd041b5a37e593e23b06c7f (127.10.33.66:38385)
I20260927 15:18:10.478541 30715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.471106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:38385 (local address 127.10.33.65:37725)
0927 15:18:10.471982 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:10.472005 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:10.472136 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:10.472486 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:10.472490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:10.472515 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:10.472937 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:10.472947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.474020 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.474024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:10.476007 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:10.476022 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.477666 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.477673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:10.477845 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:10.477907 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 15:18:10.478009 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 15:18:10.478158 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":134,"thread_start_us":743,"threads_started":1}
I20260927 15:18:10.478660 30700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.471275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:44027 (local address 127.10.33.67:34933)
0927 15:18:10.471479 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:10.471485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:10.472302 (+ 817us) server_negotiation.cc:407] Connection header received
0927 15:18:10.472449 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:10.472454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:10.472527 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:10.472655 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:10.473243 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.474040 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.476307 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.477600 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.477740 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:10.477787 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:10.478273 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 15:18:10.478429 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 15:18:10.479415 30714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.471257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.67:34933 (local address 127.10.33.65:44027)
0927 15:18:10.472195 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:10.472212 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:10.472337 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:10.472669 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:10.472672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:10.472692 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:10.473073 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:10.473084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.474195 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.474201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:10.476056 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:10.476077 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.477931 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.477936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:10.478096 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:10.478129 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:18:10.478882 (+ 753us) client_negotiation.cc:240] Negotiation successful
0927 15:18:10.479021 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":22,"thread_start_us":387,"threads_started":1}
I20260927 15:18:10.479991 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.471461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.65:37725 (local address 127.10.33.66:38385)
0927 15:18:10.471679 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:10.471686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:10.472086 (+ 400us) server_negotiation.cc:407] Connection header received
0927 15:18:10.472277 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:10.472283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:10.472370 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:10.472457 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:10.473084 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.473878 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.476218 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.477439 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.477622 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:10.477697 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:10.478052 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 15:18:10.479743 (+ 1691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:18:10.480659 30642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5d45e8953d426aa2de78a7461d64da" candidate_uuid: "bcba7828df294639bdb0a503a03ac852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "104a07fb9b3141c98e675e0b15390fc1" is_pre_election: true
I20260927 15:18:10.481166 30642 raft_consensus.cc:2454] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcba7828df294639bdb0a503a03ac852 in term 0.
I20260927 15:18:10.481429 30512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5d45e8953d426aa2de78a7461d64da" candidate_uuid: "bcba7828df294639bdb0a503a03ac852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "534b336b1dd041b5a37e593e23b06c7f" is_pre_election: true
I20260927 15:18:10.481989 30512 raft_consensus.cc:2454] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcba7828df294639bdb0a503a03ac852 in term 0.
I20260927 15:18:10.481993 30316 leader_election.cc:302] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 104a07fb9b3141c98e675e0b15390fc1, bcba7828df294639bdb0a503a03ac852; no voters:
I20260927 15:18:10.482471 30711 raft_consensus.cc:2781] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:18:10.482613 30711 raft_consensus.cc:482] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:18:10.482709 30711 raft_consensus.cc:3037] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:10.486188 30703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.479546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:42777 (local address 127.10.33.65:37137)
0927 15:18:10.479702 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:10.479708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:10.479725 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:18:10.479797 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:10.479801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:10.479890 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:10.479974 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:10.480597 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.481560 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.483942 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.485232 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.485386 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:10.485458 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:10.485739 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 15:18:10.485897 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 15:18:10.486183 30689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.478466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:37137 (local address 127.10.33.67:42777)
0927 15:18:10.479102 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:10.479120 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:10.479230 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:10.480181 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:10.480186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:10.480222 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:10.480418 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:10.480429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.481724 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.481729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:10.483716 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:10.483746 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.485370 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.485377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:10.485564 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:10.485589 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:18:10.485697 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 15:18:10.485916 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513}
I20260927 15:18:10.485920 30711 raft_consensus.cc:504] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104a07fb9b3141c98e675e0b15390fc1" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34933 } } peers { permanent_uuid: "bcba7828df294639bdb0a503a03ac852" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37137 } } peers { permanent_uuid: "534b336b1dd041b5a37e593e23b06c7f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 38385 } }
I20260927 15:18:10.487251 30381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5d45e8953d426aa2de78a7461d64da" candidate_uuid: "104a07fb9b3141c98e675e0b15390fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcba7828df294639bdb0a503a03ac852" is_pre_election: true
I20260927 15:18:10.487744 30381 raft_consensus.cc:2379] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 104a07fb9b3141c98e675e0b15390fc1 in current term 1: Already voted for candidate bcba7828df294639bdb0a503a03ac852 in this term.
I20260927 15:18:10.488008 30512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5d45e8953d426aa2de78a7461d64da" candidate_uuid: "bcba7828df294639bdb0a503a03ac852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "534b336b1dd041b5a37e593e23b06c7f"
I20260927 15:18:10.488240 30512 raft_consensus.cc:3037] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:10.488519 30711 leader_election.cc:288] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 [CANDIDATE]: Term 1 election: Requested vote from peers 104a07fb9b3141c98e675e0b15390fc1 (127.10.33.67:34933), 534b336b1dd041b5a37e593e23b06c7f (127.10.33.66:38385)
I20260927 15:18:10.488495 30642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5d45e8953d426aa2de78a7461d64da" candidate_uuid: "bcba7828df294639bdb0a503a03ac852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "104a07fb9b3141c98e675e0b15390fc1"
I20260927 15:18:10.488709 30642 raft_consensus.cc:3037] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:10.490926 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.479174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.33.67:60421 (local address 127.10.33.66:38385)
0927 15:18:10.480566 (+ 1392us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:10.480571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:10.483870 (+ 3299us) server_negotiation.cc:407] Connection header received
0927 15:18:10.484193 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:10.484199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:10.484274 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:10.484333 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:10.485167 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.486262 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.488473 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.489839 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.490022 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:10.490099 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:10.490486 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 15:18:10.490625 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1295,"thread_start_us":1730,"threads_started":1}
I20260927 15:18:10.490976 30716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.478880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.66:38385 (local address 127.10.33.67:60421)
0927 15:18:10.483786 (+ 4906us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:10.483804 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:10.484044 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:10.484575 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:10.484579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:10.484606 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:10.484995 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:10.485007 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.486451 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.486456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:10.488281 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:10.488295 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.490033 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.490040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:10.490213 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:10.490256 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 15:18:10.490459 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 15:18:10.490611 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4770,"thread_start_us":5670,"threads_started":1}
I20260927 15:18:10.491714 30512 raft_consensus.cc:2454] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcba7828df294639bdb0a503a03ac852 in term 1.
I20260927 15:18:10.491808 30642 raft_consensus.cc:2454] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcba7828df294639bdb0a503a03ac852 in term 1.
I20260927 15:18:10.491822 30511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5d45e8953d426aa2de78a7461d64da" candidate_uuid: "104a07fb9b3141c98e675e0b15390fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "534b336b1dd041b5a37e593e23b06c7f" is_pre_election: true
I20260927 15:18:10.492118 30315 leader_election.cc:302] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 534b336b1dd041b5a37e593e23b06c7f, bcba7828df294639bdb0a503a03ac852; no voters:
I20260927 15:18:10.492221 30511 raft_consensus.cc:2379] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 104a07fb9b3141c98e675e0b15390fc1 in current term 1: Already voted for candidate bcba7828df294639bdb0a503a03ac852 in this term.
I20260927 15:18:10.492470 30711 raft_consensus.cc:2781] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:10.493021 30576 leader_election.cc:302] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 104a07fb9b3141c98e675e0b15390fc1; no voters: 534b336b1dd041b5a37e593e23b06c7f, bcba7828df294639bdb0a503a03ac852
I20260927 15:18:10.493484 30713 raft_consensus.cc:2726] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 15:18:10.493973 30711 raft_consensus.cc:686] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 [term 1 LEADER]: Becoming Leader. State: Replica: bcba7828df294639bdb0a503a03ac852, State: Running, Role: LEADER
I20260927 15:18:10.494458 30711 consensus_queue.cc:237] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104a07fb9b3141c98e675e0b15390fc1" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34933 } } peers { permanent_uuid: "bcba7828df294639bdb0a503a03ac852" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37137 } } peers { permanent_uuid: "534b336b1dd041b5a37e593e23b06c7f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 38385 } }
I20260927 15:18:10.499230 30238 catalog_manager.cc:5495] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcba7828df294639bdb0a503a03ac852 (127.10.33.65). New cstate: current_term: 1 leader_uuid: "bcba7828df294639bdb0a503a03ac852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104a07fb9b3141c98e675e0b15390fc1" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcba7828df294639bdb0a503a03ac852" member_type: VOTER last_known_addr { host: "127.10.33.65" port: 37137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "534b336b1dd041b5a37e593e23b06c7f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 38385 } health_report { overall_health: UNKNOWN } } }
W20260927 15:18:10.503116 30688 tablet.cc:2369] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:18:10.530882 30558 tablet.cc:2369] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:18:10.533664 30703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.528161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37604 (local address 127.10.33.65:37137)
0927 15:18:10.528336 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:10.528340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:10.528425 (+ 85us) server_negotiation.cc:407] Connection header received
0927 15:18:10.528597 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:10.528601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:10.528723 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:10.528866 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:10.529262 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.530129 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.531551 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.532029 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.532127 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:10.532620 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:10.533137 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:10.533249 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:10.533350 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 15:18:10.533418 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:18:10.541723 30642 raft_consensus.cc:1261] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1 [term 1 FOLLOWER]: Refusing update from remote peer bcba7828df294639bdb0a503a03ac852: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:18:10.541731 30511 raft_consensus.cc:1261] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f [term 1 FOLLOWER]: Refusing update from remote peer bcba7828df294639bdb0a503a03ac852: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:18:10.542665 30718 consensus_queue.cc:1035] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "104a07fb9b3141c98e675e0b15390fc1" member_type: VOTER last_known_addr { host: "127.10.33.67" port: 34933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:18:10.543026 30711 consensus_queue.cc:1035] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "534b336b1dd041b5a37e593e23b06c7f" member_type: VOTER last_known_addr { host: "127.10.33.66" port: 38385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:18:10.554013 30721 mvcc.cc:205] Tried to move back new op lower bound from 7333979302049251328 to 7333979301868765184. Current Snapshot: MvccSnapshot[applied={T|T < 7333979302049251328}]
I20260927 15:18:10.568693 30240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:18:10.568964 30240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:18:10.573006 30240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0da698a8c79f41d885b9b337e28c1ff9: Sending DeleteTablet for 3 replicas of tablet 9a5d45e8953d426aa2de78a7461d64da
I20260927 15:18:10.574383 30361 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a5d45e8953d426aa2de78a7461d64da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:18:10 UTC) from {username='slave'} at 127.0.0.1:37600
I20260927 15:18:10.574375 30622 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a5d45e8953d426aa2de78a7461d64da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:18:10 UTC) from {username='slave'} at 127.0.0.1:38420
I20260927 15:18:10.574538 30492 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a5d45e8953d426aa2de78a7461d64da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:18:10 UTC) from {username='slave'} at 127.0.0.1:33964
I20260927 15:18:10.576627 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 30299
I20260927 15:18:10.585513 30732 tablet_replica.cc:330] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852: stopping tablet replica
I20260927 15:18:10.586252 30732 raft_consensus.cc:2229] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:18:10.587298 30732 raft_consensus.cc:2258] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:18:10.587999 30731 tablet_replica.cc:330] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1: stopping tablet replica
I20260927 15:18:10.588896 30731 raft_consensus.cc:2229] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:18:10.591240 30731 raft_consensus.cc:2258] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:18:10.593096 30732 ts_tablet_manager.cc:1905] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:18:10.594646 30731 ts_tablet_manager.cc:1905] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:18:10.600034 30733 tablet_replica.cc:330] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f: stopping tablet replica
I20260927 15:18:10.600731 30703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.578118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37612 (local address 127.10.33.65:37137)
0927 15:18:10.578292 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:10.578298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:10.592182 (+ 13884us) server_negotiation.cc:407] Connection header received
0927 15:18:10.592457 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:10.592463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:10.592587 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:10.592710 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:10.593163 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.594216 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.596013 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.596726 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.596840 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:10.598316 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:10.598326 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:10.598343 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:10.599146 (+ 803us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:10.599207 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:10.599232 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:10.599311 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:10.600233 (+ 922us) server_negotiation.cc:299] Negotiation successful
0927 15:18:10.600360 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:18:10.601725 30733 raft_consensus.cc:2229] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:18:10.601995 30421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:10.602458 30733 raft_consensus.cc:2258] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:18:10.604913 30733 ts_tablet_manager.cc:1905] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:18:10.609987 30731 ts_tablet_manager.cc:1918] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:18:10.610177 30731 log.cc:1196] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-2/wal/wals/9a5d45e8953d426aa2de78a7461d64da
I20260927 15:18:10.610700 30731 ts_tablet_manager.cc:1939] T 9a5d45e8953d426aa2de78a7461d64da P 104a07fb9b3141c98e675e0b15390fc1: Deleting consensus metadata
I20260927 15:18:10.611698 30733 ts_tablet_manager.cc:1918] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:18:10.611867 30733 log.cc:1196] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-1/wal/wals/9a5d45e8953d426aa2de78a7461d64da
I20260927 15:18:10.612378 30733 ts_tablet_manager.cc:1939] T 9a5d45e8953d426aa2de78a7461d64da P 534b336b1dd041b5a37e593e23b06c7f: Deleting consensus metadata
I20260927 15:18:10.612700 30226 catalog_manager.cc:4841] TS 104a07fb9b3141c98e675e0b15390fc1 (127.10.33.67:34933): tablet 9a5d45e8953d426aa2de78a7461d64da (table test-table [id=d6422b4b1c5640b4925867383edb1fff]) successfully deleted
I20260927 15:18:10.613956 30225 catalog_manager.cc:4841] TS 534b336b1dd041b5a37e593e23b06c7f (127.10.33.66:38385): tablet 9a5d45e8953d426aa2de78a7461d64da (table test-table [id=d6422b4b1c5640b4925867383edb1fff]) successfully deleted
I20260927 15:18:10.702009 30732 ts_tablet_manager.cc:1918] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:18:10.702440 30732 log.cc:1196] T 9a5d45e8953d426aa2de78a7461d64da P bcba7828df294639bdb0a503a03ac852: Deleting WAL directory at /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521968392480-10373-0/minicluster-data/ts-0/wal/wals/9a5d45e8953d426aa2de78a7461d64da
I20260927 15:18:10.702852 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 30299
I20260927 15:18:10.718012 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 30430
W20260927 15:18:10.718078 30227 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.33.65:37137 (error 108)
I20260927 15:18:10.720785 30699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.720298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:37137 (local address 127.0.0.1:37626)
0927 15:18:10.720472 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:10.720603 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:37137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 15:18:10.721042 30227 catalog_manager.cc:4584] TS bcba7828df294639bdb0a503a03ac852 (127.10.33.65:37137): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a5d45e8953d426aa2de78a7461d64da: Network error: Client connection negotiation failed: client connection to 127.10.33.65:37137: connect: Connection refused (error 111)
I20260927 15:18:10.728528 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.718882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.10.33.66:38385)
0927 15:18:10.719097 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:10.719104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:10.719126 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:18:10.719213 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:10.719219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:10.719341 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:10.719454 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:10.720095 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.721065 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.722448 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.722867 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.722976 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:10.727119 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:10.727129 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:10.727181 (+ 52us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:10.727749 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:10.727801 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:10.727823 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:10.727924 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:10.728092 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 15:18:10.728230 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:18:10.729354 30552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:10.774930 30699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.774291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:37137 (local address 127.0.0.1:37640)
0927 15:18:10.774589 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:10.774727 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:37137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 15:18:10.776075 30699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.775654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:37137 (local address 127.0.0.1:37654)
0927 15:18:10.775831 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:10.775928 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:37137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 15:18:10.818127 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 30430
I20260927 15:18:10.832077 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 30560
I20260927 15:18:10.836733 30699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.836187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:37137 (local address 127.0.0.1:37668)
0927 15:18:10.836379 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:10.836517 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:37137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 15:18:10.837998 30699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.837492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:37137 (local address 127.0.0.1:37682)
0927 15:18:10.837675 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:10.837780 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:37137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 15:18:10.839612 30700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.832900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38428 (local address 127.10.33.67:34933)
0927 15:18:10.833125 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:10.833131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:10.833154 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:18:10.833290 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:10.833297 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:10.833393 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:10.833495 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:10.833963 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.834770 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.836231 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:10.836842 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:10.836985 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:10.838215 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:10.838227 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:10.838243 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:10.838877 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:10.838930 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:10.838953 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:10.839057 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:10.839196 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 15:18:10.839303 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 15:18:10.840344 30682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:10.919015 30699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.918344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:37137 (local address 127.0.0.1:37694)
0927 15:18:10.918647 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:10.918777 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:37137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 15:18:10.920426 30699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:10.919843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.65:37137 (local address 127.0.0.1:37700)
0927 15:18:10.920093 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:10.920199 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.65:37137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260927 15:18:10.931705 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 30560
I20260927 15:18:10.945983 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 30209
I20260927 15:18:10.947232 30270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:11.055526 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 30209
2026-09-27T15:18:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2834 ms)
[----------] 6 tests from EncryptionPolicyTest (21414 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:18:11 dist-test-slave-f5h6 krb5kdc[30742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:18:11 dist-test-slave-f5h6 krb5kdc[30742](info): set up 2 sockets
Sep 27 15:18:11 dist-test-slave-f5h6 krb5kdc[30742](info): commencing operation
krb5kdc: starting...
W20260927 15:18:12.122406 10373 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:18:12 dist-test-slave-f5h6 krb5kdc[30742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:18:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:18:12.370585 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46791
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:12.700767 30755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:12.701113 30755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:12.701256 30755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:12.708918 30755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:12.709079 30755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:12.709151 30755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:12.709225 30755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:12.709286 30755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:12.719973 30755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46791
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:12.721328 30755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:12.722648 30755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:12.734280 30760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:12.734213 30763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:12.734249 30761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:12.735414 30755 server_base.cc:1034] running on GCE node
I20260927 15:18:12.736233 30755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:12.737715 30755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:12.738947 30755 hybrid_clock.cc:648] HybridClock initialized: now 1790522292738885 us; error 49 us; skew 500 ppm
I20260927 15:18:12.739323 30755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:12 dist-test-slave-f5h6 krb5kdc[30742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:12.742512 30755 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:18:12.744042 30755 webserver.cc:460] Webserver started at http://127.0.0.1:44507/ using document root <none> and password file <none>
I20260927 15:18:12.744702 30755 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:12.744818 30755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:12.745110 30755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:12.748099 30755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "e90dfcd029354e49b0292b6ce4f50cdc"
format_stamp: "Formatted at 2026-09-27 15:18:12 on dist-test-slave-f5h6"
server_key: "6cb6c8281bc4c1b55f7c9cf09b3b72cd"
server_key_iv: "014c8796591fa37c1a84bda2807e15e3"
server_key_version: "encryptionkey@0"
I20260927 15:18:12.748847 30755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "e90dfcd029354e49b0292b6ce4f50cdc"
format_stamp: "Formatted at 2026-09-27 15:18:12 on dist-test-slave-f5h6"
server_key: "6cb6c8281bc4c1b55f7c9cf09b3b72cd"
server_key_iv: "014c8796591fa37c1a84bda2807e15e3"
server_key_version: "encryptionkey@0"
I20260927 15:18:12.754129 30755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260927 15:18:12.757735 30770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:12.759248 30755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:18:12.759435 30755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "e90dfcd029354e49b0292b6ce4f50cdc"
format_stamp: "Formatted at 2026-09-27 15:18:12 on dist-test-slave-f5h6"
server_key: "6cb6c8281bc4c1b55f7c9cf09b3b72cd"
server_key_iv: "014c8796591fa37c1a84bda2807e15e3"
server_key_version: "encryptionkey@0"
I20260927 15:18:12.759629 30755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:12.785117 30755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:12.789494 30755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:12.789767 30755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:12.807828 30755 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46791
I20260927 15:18:12.807859 30821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46791 every 8 connection(s)
I20260927 15:18:12.809234 30755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:12.813056 30822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:12.815644 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 30755
I20260927 15:18:12.815850 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:12.816601 10373 external_mini_cluster.cc:1428] Setting key 469ce20231eeeb9f7556b6dab11158e7
I20260927 15:18:12.825456 30822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc: Bootstrap starting.
I20260927 15:18:12.829344 30822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:12.830722 30822 log.cc:824] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:18:12 dist-test-slave-f5h6 krb5kdc[30742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:18:12.834206 30822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc: No bootstrap required, opened a new log
I20260927 15:18:12.841372 30825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:12.818813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56947 (local address 127.0.0.1:46791)
0927 15:18:12.819568 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:12.819579 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:12.819627 (+ 48us) server_negotiation.cc:407] Connection header received
0927 15:18:12.821262 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:12.821284 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:12.821718 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:12.822096 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:12.823825 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:12.825176 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:12.826939 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:12.827726 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:12.827877 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:12.833008 (+ 5131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:12.833017 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:12.833033 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:12.833076 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:12.836429 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:12.837531 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:12.837539 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:12.837546 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:12.837740 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:12.838328 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:12.838332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:12.838334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:12.838914 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:12.839172 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:12.839761 (+ 589us) server_negotiation.cc:299] Negotiation successful
0927 15:18:12.840216 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":349,"threads_started":1}
I20260927 15:18:12.843513 30822 raft_consensus.cc:348] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90dfcd029354e49b0292b6ce4f50cdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46791 } }
I20260927 15:18:12.843918 30822 raft_consensus.cc:374] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:12.844026 30822 raft_consensus.cc:729] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90dfcd029354e49b0292b6ce4f50cdc, State: Initialized, Role: FOLLOWER
I20260927 15:18:12.844925 30822 consensus_queue.cc:260] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90dfcd029354e49b0292b6ce4f50cdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46791 } }
I20260927 15:18:12.845273 30822 raft_consensus.cc:388] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:18:12.845423 30822 raft_consensus.cc:482] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:18:12.845592 30822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:12.846021 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:18:12.847913 30822 raft_consensus.cc:504] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90dfcd029354e49b0292b6ce4f50cdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46791 } }
I20260927 15:18:12.848403 30822 leader_election.cc:302] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e90dfcd029354e49b0292b6ce4f50cdc; no voters:
I20260927 15:18:12.849018 30822 leader_election.cc:288] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:18:12.849197 30829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:12.850322 30829 raft_consensus.cc:686] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc [term 1 LEADER]: Becoming Leader. State: Replica: e90dfcd029354e49b0292b6ce4f50cdc, State: Running, Role: LEADER
I20260927 15:18:12.851039 30829 consensus_queue.cc:237] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90dfcd029354e49b0292b6ce4f50cdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46791 } }
I20260927 15:18:12.851475 30822 sys_catalog.cc:564] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:12.855101 30832 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e90dfcd029354e49b0292b6ce4f50cdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90dfcd029354e49b0292b6ce4f50cdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46791 } } }
I20260927 15:18:12.855536 30832 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:12.856212 30833 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e90dfcd029354e49b0292b6ce4f50cdc. Latest consensus state: current_term: 1 leader_uuid: "e90dfcd029354e49b0292b6ce4f50cdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90dfcd029354e49b0292b6ce4f50cdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46791 } } }
I20260927 15:18:12.856582 30833 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:12.861073 30838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:12.866916 30838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:12.870729 30825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:12.850840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54765 (local address 127.0.0.1:46791)
0927 15:18:12.851151 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:12.851161 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:12.851781 (+ 620us) server_negotiation.cc:407] Connection header received
0927 15:18:12.851994 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:12.852003 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:12.852070 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:12.852154 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:12.853732 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:12.854916 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:12.856788 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:12.857536 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:12.857661 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:12.861166 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:12.861174 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:12.861179 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:12.861237 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:12.865778 (+ 4541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:12.867117 (+ 1339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:12.867124 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:12.867128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:12.867310 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:12.867945 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:12.867951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:12.867955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:12.868475 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:12.869894 (+ 1419us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:12.870102 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 15:18:12.870323 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 15:18:12.878120 30838 catalog_manager.cc:1347] Generated new cluster ID: 63bbf95127ae482db16007359e55beb2
I20260927 15:18:12.878247 30838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:12.889688 30838 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:12.891137 30838 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:12.899443 30838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e90dfcd029354e49b0292b6ce4f50cdc: Generated new TSK 0
I20260927 15:18:12.900411 30838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:18:12.906452 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 30755
I20260927 15:18:12.908342 30817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:12.998592 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 30755
W20260927 15:18:13.012923 30740 connection.cc:535] client connection to 127.0.0.1:46791 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46791): Connection reset by peer (error 104)
2026-09-27T15:18:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1925 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:18:13 dist-test-slave-f5h6 krb5kdc[30855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:18:13 dist-test-slave-f5h6 krb5kdc[30855](info): set up 2 sockets
Sep 27 15:18:13 dist-test-slave-f5h6 krb5kdc[30855](info): commencing operation
krb5kdc: starting...
W20260927 15:18:14.033960 10373 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:18:14 dist-test-slave-f5h6 krb5kdc[30855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:18:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:18:14.290289 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38477
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:14.632743 30868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:14.633040 30868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:14.633147 30868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:14.640327 30868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:14.640456 30868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:14.640501 30868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:14.640561 30868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:14.640604 30868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:14.650748 30868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38477
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:14.652051 30868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:14.653370 30868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:14.664798 30874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:14.664820 30873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:14.664852 30876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:14.665557 30868 server_base.cc:1034] running on GCE node
I20260927 15:18:14.666375 30868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:14.667774 30868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:14.668974 30868 hybrid_clock.cc:648] HybridClock initialized: now 1790522294668932 us; error 37 us; skew 500 ppm
I20260927 15:18:14.669310 30868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:14 dist-test-slave-f5h6 krb5kdc[30855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:14.672389 30868 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:18:14.673954 30868 webserver.cc:460] Webserver started at http://127.0.0.1:35677/ using document root <none> and password file <none>
I20260927 15:18:14.674592 30868 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:14.674693 30868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:14.674943 30868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:14.678020 30868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "2f0ea8acb8ef4303b927de724e7892a3"
format_stamp: "Formatted at 2026-09-27 15:18:14 on dist-test-slave-f5h6"
server_key: "01c198a3153225f676b5bcbf799c4726"
server_key_iv: "cf44bb0babba285fcb4f585b41bda387"
server_key_version: "encryptionkey@0"
I20260927 15:18:14.678790 30868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "2f0ea8acb8ef4303b927de724e7892a3"
format_stamp: "Formatted at 2026-09-27 15:18:14 on dist-test-slave-f5h6"
server_key: "01c198a3153225f676b5bcbf799c4726"
server_key_iv: "cf44bb0babba285fcb4f585b41bda387"
server_key_version: "encryptionkey@0"
I20260927 15:18:14.684182 30868 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:18:14.688067 30883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:14.689999 30868 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:18:14.690218 30868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "2f0ea8acb8ef4303b927de724e7892a3"
format_stamp: "Formatted at 2026-09-27 15:18:14 on dist-test-slave-f5h6"
server_key: "01c198a3153225f676b5bcbf799c4726"
server_key_iv: "cf44bb0babba285fcb4f585b41bda387"
server_key_version: "encryptionkey@0"
I20260927 15:18:14.690438 30868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:14.704670 30868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:14.709825 30868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:14.710162 30868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:14.728346 30868 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38477
I20260927 15:18:14.728371 30934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38477 every 8 connection(s)
I20260927 15:18:14.729756 30868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:14.733633 30935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:14.734841 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 30868
I20260927 15:18:14.735031 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:14.735581 10373 external_mini_cluster.cc:1428] Setting key 2bebb2893f180fdc5c9f969553b66d0c
I20260927 15:18:14.747524 30935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3: Bootstrap starting.
I20260927 15:18:14.749708 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:14.737396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58817 (local address 127.0.0.1:38477)
0927 15:18:14.738296 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:14.738307 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:14.738352 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:18:14.739980 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:14.740007 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:14.740443 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:14.740794 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:14.741364 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:14.742739 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:14.744562 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:14.745358 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:14.745491 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:14.746950 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:14.746960 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:14.746982 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:14.747703 (+ 721us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:14.747811 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:14.747843 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:14.747995 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:14.748399 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 15:18:14.748603 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":428,"threads_started":1}
I20260927 15:18:14.751092 30935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:14.752141 30935 log.cc:824] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:14.753681 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:18:14.755533 30935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3: No bootstrap required, opened a new log
I20260927 15:18:14.765882 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:14.758854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60173 (local address 127.0.0.1:38477)
0927 15:18:14.759047 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:14.759054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:14.759469 (+ 415us) server_negotiation.cc:407] Connection header received
0927 15:18:14.759614 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:14.759620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:14.759707 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:14.759824 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:14.760364 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:14.761542 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:14.763088 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:14.763807 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:14.763951 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:14.765040 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:14.765047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:14.765051 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:14.765075 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:14.765126 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:14.765147 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:14.765270 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:14.765435 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 15:18:14.765491 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:18:14.765573 30935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0ea8acb8ef4303b927de724e7892a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38477 } }
I20260927 15:18:14.766150 30935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:14.766251 30935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f0ea8acb8ef4303b927de724e7892a3, State: Initialized, Role: FOLLOWER
I20260927 15:18:14.766984 30935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0ea8acb8ef4303b927de724e7892a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38477 } }
I20260927 15:18:14.767294 30935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:18:14.767434 30935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:18:14.767590 30935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:14.769821 30935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0ea8acb8ef4303b927de724e7892a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38477 } }
I20260927 15:18:14.770418 30935 leader_election.cc:302] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f0ea8acb8ef4303b927de724e7892a3; no voters:
I20260927 15:18:14.771132 30935 leader_election.cc:288] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:18:14.771346 30945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:14.772511 30945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3 [term 1 LEADER]: Becoming Leader. State: Replica: 2f0ea8acb8ef4303b927de724e7892a3, State: Running, Role: LEADER
I20260927 15:18:14.773101 30945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0ea8acb8ef4303b927de724e7892a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38477 } }
I20260927 15:18:14.773437 30935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:14.776175 30947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f0ea8acb8ef4303b927de724e7892a3. Latest consensus state: current_term: 1 leader_uuid: "2f0ea8acb8ef4303b927de724e7892a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0ea8acb8ef4303b927de724e7892a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38477 } } }
I20260927 15:18:14.776147 30946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f0ea8acb8ef4303b927de724e7892a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0ea8acb8ef4303b927de724e7892a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38477 } } }
I20260927 15:18:14.776662 30946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:14.776662 30947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:14.785337 30952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:14.789763 30952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:14.798856 30952 catalog_manager.cc:1347] Generated new cluster ID: 5fc3e63d0a874b45b2f668d18a0002c5
I20260927 15:18:14.799002 30952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:14.824956 30952 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:14.826963 30952 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:14.835909 30952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f0ea8acb8ef4303b927de724e7892a3: Generated new TSK 0
I20260927 15:18:14.837052 30952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:18:14.846767 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 30868
W20260927 15:18:14.848426 30930 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58817
W20260927 15:18:14.849324 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 30868: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:18:14.849509 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 30868
W20260927 15:18:14.863961 30853 connection.cc:535] client connection to 127.0.0.1:38477 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38477): Connection reset by peer (error 104)
2026-09-27T15:18:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1851 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:18:14 dist-test-slave-f5h6 krb5kdc[30966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:18:14 dist-test-slave-f5h6 krb5kdc[30966](info): set up 2 sockets
Sep 27 15:18:14 dist-test-slave-f5h6 krb5kdc[30966](info): commencing operation
krb5kdc: starting...
W20260927 15:18:15.878211 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:18:16 dist-test-slave-f5h6 krb5kdc[30966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:18:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:18:16.145537 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43183
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:16.481020 30979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:16.481348 30979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:16.481498 30979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:16.489449 30979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:16.489614 30979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:16.489692 30979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:16.489776 30979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:16.489842 30979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:16.500247 30979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43183
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:16.501607 30979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:16.502929 30979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:16.515869 30984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:16.515921 30985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:16.515974 30987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:16.516659 30979 server_base.cc:1034] running on GCE node
I20260927 15:18:16.517488 30979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:16.519094 30979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:16.520350 30979 hybrid_clock.cc:648] HybridClock initialized: now 1790522296520296 us; error 40 us; skew 500 ppm
I20260927 15:18:16.520737 30979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:16 dist-test-slave-f5h6 krb5kdc[30966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:16.524106 30979 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:18:16.526062 30979 webserver.cc:460] Webserver started at http://127.0.0.1:38625/ using document root <none> and password file <none>
I20260927 15:18:16.526806 30979 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:16.526934 30979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:16.527228 30979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:16.530541 30979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "74ddbe6a62824bc688879e4ae819e3fd"
format_stamp: "Formatted at 2026-09-27 15:18:16 on dist-test-slave-f5h6"
server_key: "7ab1405b97fb4a87235b3c521573c2af"
server_key_iv: "82afaed95a4f4c17cfd86014d334012c"
server_key_version: "encryptionkey@0"
I20260927 15:18:16.531347 30979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "74ddbe6a62824bc688879e4ae819e3fd"
format_stamp: "Formatted at 2026-09-27 15:18:16 on dist-test-slave-f5h6"
server_key: "7ab1405b97fb4a87235b3c521573c2af"
server_key_iv: "82afaed95a4f4c17cfd86014d334012c"
server_key_version: "encryptionkey@0"
I20260927 15:18:16.536871 30979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 15:18:16.540553 30994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:16.542078 30979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:18:16.542317 30979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "74ddbe6a62824bc688879e4ae819e3fd"
format_stamp: "Formatted at 2026-09-27 15:18:16 on dist-test-slave-f5h6"
server_key: "7ab1405b97fb4a87235b3c521573c2af"
server_key_iv: "82afaed95a4f4c17cfd86014d334012c"
server_key_version: "encryptionkey@0"
I20260927 15:18:16.542543 30979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:16.548892 30979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:16.553359 30979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:16.553660 30979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:16.571702 30979 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43183
I20260927 15:18:16.571771 31045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43183 every 8 connection(s)
I20260927 15:18:16.573109 30979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:16.576913 31046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:16.580533 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 30979
I20260927 15:18:16.580729 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:16.581238 10373 external_mini_cluster.cc:1428] Setting key 509b6a71bdd160ad097116783f59e885
I20260927 15:18:16.589847 31049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:16.583326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33741 (local address 127.0.0.1:43183)
0927 15:18:16.584275 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:16.584288 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:16.584333 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:18:16.586147 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:16.586181 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:16.586585 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:16.587010 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:16.588110 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:16.588117 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:16.588131 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:16.588148 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:16.588220 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:16.588241 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:16.588466 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:16.588616 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 15:18:16.588826 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":432,"threads_started":1}
I20260927 15:18:16.590121 31046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd: Bootstrap starting.
I20260927 15:18:16.593847 31046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:16.594408 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:18:16.595106 31046 log.cc:824] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:16.598726 31046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd: No bootstrap required, opened a new log
I20260927 15:18:16.602144 31049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:16.599190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48881 (local address 127.0.0.1:43183)
0927 15:18:16.599393 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:16.599400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:16.599953 (+ 553us) server_negotiation.cc:407] Connection header received
0927 15:18:16.600224 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:16.600228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:16.600289 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:16.600387 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:16.601404 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:16.601408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:16.601414 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:16.601432 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:16.601476 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:16.601495 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:16.601618 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:16.601732 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 15:18:16.601789 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:18:16.607275 31046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74ddbe6a62824bc688879e4ae819e3fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43183 } }
I20260927 15:18:16.607712 31046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:16.607813 31046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74ddbe6a62824bc688879e4ae819e3fd, State: Initialized, Role: FOLLOWER
I20260927 15:18:16.608409 31046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74ddbe6a62824bc688879e4ae819e3fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43183 } }
I20260927 15:18:16.608671 31046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:18:16.608810 31046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:18:16.609004 31046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:16.611420 31046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74ddbe6a62824bc688879e4ae819e3fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43183 } }
I20260927 15:18:16.611984 31046 leader_election.cc:302] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74ddbe6a62824bc688879e4ae819e3fd; no voters:
I20260927 15:18:16.612637 31046 leader_election.cc:288] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:18:16.612807 31056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:16.613559 31056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd [term 1 LEADER]: Becoming Leader. State: Replica: 74ddbe6a62824bc688879e4ae819e3fd, State: Running, Role: LEADER
I20260927 15:18:16.614148 31056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74ddbe6a62824bc688879e4ae819e3fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43183 } }
I20260927 15:18:16.614815 31046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:16.617121 31058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74ddbe6a62824bc688879e4ae819e3fd. Latest consensus state: current_term: 1 leader_uuid: "74ddbe6a62824bc688879e4ae819e3fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74ddbe6a62824bc688879e4ae819e3fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43183 } } }
I20260927 15:18:16.617202 31057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74ddbe6a62824bc688879e4ae819e3fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74ddbe6a62824bc688879e4ae819e3fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43183 } } }
I20260927 15:18:16.617571 31057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:16.617571 31058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:16.619292 31061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:16.624415 31061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:16.634124 31061 catalog_manager.cc:1347] Generated new cluster ID: 07afd7ef64ba4b85bf3aefda70b8d2ce
I20260927 15:18:16.634254 31061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:16.660805 31061 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:16.661415 31061 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:16.673703 31061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74ddbe6a62824bc688879e4ae819e3fd: Generated new TSK 0
I20260927 15:18:16.674940 31061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:18:16.689121 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 30979
W20260927 15:18:16.690385 31041 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33741
W20260927 15:18:16.691131 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 30979: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:18:16.691268 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 30979
2026-09-27T15:18:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1841 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:18:16 dist-test-slave-f5h6 krb5kdc[31077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:18:16 dist-test-slave-f5h6 krb5kdc[31077](info): set up 2 sockets
Sep 27 15:18:16 dist-test-slave-f5h6 krb5kdc[31077](info): commencing operation
krb5kdc: starting...
W20260927 15:18:17.714001 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:18:17 dist-test-slave-f5h6 krb5kdc[31077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:18:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:18:17.970878 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41917
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:18.304646 31090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:18.304968 31090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:18.305097 31090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:18.312459 31090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:18.312621 31090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:18.312705 31090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:18.312821 31090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:18.312883 31090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:18.323081 31090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:18.324411 31090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:18.325671 31090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:18.337538 31098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:18.337575 31096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:18.337653 31095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:18.338311 31090 server_base.cc:1034] running on GCE node
I20260927 15:18:18.339102 31090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:18.340523 31090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:18.341773 31090 hybrid_clock.cc:648] HybridClock initialized: now 1790522298341706 us; error 55 us; skew 500 ppm
I20260927 15:18:18.342156 31090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:18 dist-test-slave-f5h6 krb5kdc[31077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:18.345474 31090 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:18:18.347048 31090 webserver.cc:460] Webserver started at http://127.0.0.1:33463/ using document root <none> and password file <none>
I20260927 15:18:18.347728 31090 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:18.347853 31090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:18.348146 31090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:18.351567 31090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "43ba4708a1d549dfa8d5bafd99c541d7"
format_stamp: "Formatted at 2026-09-27 15:18:18 on dist-test-slave-f5h6"
server_key: "77941ff1ffbc71f680184481d6dcdd29"
server_key_iv: "bd5cd767e8eaf1d879e1d7cc390ccef2"
server_key_version: "encryptionkey@0"
I20260927 15:18:18.352460 31090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "43ba4708a1d549dfa8d5bafd99c541d7"
format_stamp: "Formatted at 2026-09-27 15:18:18 on dist-test-slave-f5h6"
server_key: "77941ff1ffbc71f680184481d6dcdd29"
server_key_iv: "bd5cd767e8eaf1d879e1d7cc390ccef2"
server_key_version: "encryptionkey@0"
I20260927 15:18:18.357897 31090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 15:18:18.361819 31105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:18.363283 31090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 15:18:18.363471 31090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "43ba4708a1d549dfa8d5bafd99c541d7"
format_stamp: "Formatted at 2026-09-27 15:18:18 on dist-test-slave-f5h6"
server_key: "77941ff1ffbc71f680184481d6dcdd29"
server_key_iv: "bd5cd767e8eaf1d879e1d7cc390ccef2"
server_key_version: "encryptionkey@0"
I20260927 15:18:18.363672 31090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:18.389889 31090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:18.394104 31090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:18.394378 31090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:18.411737 31090 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41917
I20260927 15:18:18.411772 31156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41917 every 8 connection(s)
I20260927 15:18:18.413033 31090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:18.415822 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 31090
I20260927 15:18:18.416005 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:18.416524 10373 external_mini_cluster.cc:1428] Setting key 5dbe35dbd5965bdcaa326eabfcf6f703
I20260927 15:18:18.417011 31157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:18.431458 31157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7: Bootstrap starting.
Sep 27 15:18:18 dist-test-slave-f5h6 krb5kdc[31077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:18:18.434741 31157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:18.435837 31157 log.cc:824] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:18.438607 31157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7: No bootstrap required, opened a new log
I20260927 15:18:18.439987 31160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:18.418233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49793 (local address 127.0.0.1:41917)
0927 15:18:18.418952 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:18.418963 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:18.419000 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:18:18.421435 (+ 2435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:18.421457 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:18.421896 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:18.422222 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:18.423995 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:18.425379 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:18.427226 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:18.428042 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:18.428199 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:18.432975 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:18.432984 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:18.432999 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:18.433052 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:18.436087 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:18.436787 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:18.436793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:18.436800 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:18.436974 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:18.437366 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:18.437369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:18.437371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:18.437935 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:18.438209 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:18.438653 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 15:18:18.439056 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":331,"threads_started":1}
I20260927 15:18:18.443465 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:18:18.446409 31157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ba4708a1d549dfa8d5bafd99c541d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41917 } }
I20260927 15:18:18.446770 31157 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:18.446863 31157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ba4708a1d549dfa8d5bafd99c541d7, State: Initialized, Role: FOLLOWER
I20260927 15:18:18.447399 31157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ba4708a1d549dfa8d5bafd99c541d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41917 } }
I20260927 15:18:18.447665 31157 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:18:18.447853 31157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:18:18.448024 31157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:18.450675 31157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ba4708a1d549dfa8d5bafd99c541d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41917 } }
I20260927 15:18:18.451333 31157 leader_election.cc:302] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43ba4708a1d549dfa8d5bafd99c541d7; no voters:
I20260927 15:18:18.451958 31157 leader_election.cc:288] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:18:18.452069 31167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:18.453193 31167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7 [term 1 LEADER]: Becoming Leader. State: Replica: 43ba4708a1d549dfa8d5bafd99c541d7, State: Running, Role: LEADER
I20260927 15:18:18.453845 31167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ba4708a1d549dfa8d5bafd99c541d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41917 } }
I20260927 15:18:18.454411 31157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:18.458732 31168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43ba4708a1d549dfa8d5bafd99c541d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ba4708a1d549dfa8d5bafd99c541d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41917 } } }
I20260927 15:18:18.459128 31168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:18.469058 31169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43ba4708a1d549dfa8d5bafd99c541d7. Latest consensus state: current_term: 1 leader_uuid: "43ba4708a1d549dfa8d5bafd99c541d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ba4708a1d549dfa8d5bafd99c541d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41917 } } }
I20260927 15:18:18.470160 31169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:18.470970 31174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:18.471000 31160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:18.447663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56021 (local address 127.0.0.1:41917)
0927 15:18:18.447843 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:18.447851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:18.448273 (+ 422us) server_negotiation.cc:407] Connection header received
0927 15:18:18.448400 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:18.448405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:18.448481 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:18.448592 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:18.450123 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:18.451125 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:18.452682 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:18.453391 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:18.453522 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:18.456340 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:18.456348 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:18.456353 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:18.456391 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:18.466483 (+ 10092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:18.467792 (+ 1309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:18.467799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:18.467803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:18.467983 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:18.468423 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:18.468427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:18.468430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:18.469177 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:18.470069 (+ 892us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:18.470247 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 15:18:18.470506 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:18:18.477262 31174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:18.487295 31174 catalog_manager.cc:1347] Generated new cluster ID: 8cc57e57e4004647896b44a06adfba56
I20260927 15:18:18.487444 31174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:18.499874 31174 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:18.501711 31174 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:18.517248 31174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43ba4708a1d549dfa8d5bafd99c541d7: Generated new TSK 0
I20260927 15:18:18.518219 31174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:18:18.530586 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 31090
I20260927 15:18:18.532130 31152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:18.623446 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 31090
2026-09-27T15:18:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1932 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:18:18 dist-test-slave-f5h6 krb5kdc[31189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:18:18 dist-test-slave-f5h6 krb5kdc[31189](info): set up 2 sockets
Sep 27 15:18:18 dist-test-slave-f5h6 krb5kdc[31189](info): commencing operation
krb5kdc: starting...
W20260927 15:18:19.642127 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:18:19 dist-test-slave-f5h6 krb5kdc[31189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:18:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:18:19.898907 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39917
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:20.241540 31202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:20.241905 31202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:20.242051 31202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:20.249485 31202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:20.249625 31202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:20.249701 31202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:20.249781 31202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:20.249848 31202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:20.260007 31202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:20.261348 31202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:20.262701 31202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:20.276207 31207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:20.276340 31210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:20.276366 31208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:20.277364 31202 server_base.cc:1034] running on GCE node
I20260927 15:18:20.278390 31202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:20.279915 31202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:20.281155 31202 hybrid_clock.cc:648] HybridClock initialized: now 1790522300281102 us; error 46 us; skew 500 ppm
I20260927 15:18:20.281538 31202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:20 dist-test-slave-f5h6 krb5kdc[31189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:20.284931 31202 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:18:20.286646 31202 webserver.cc:460] Webserver started at http://127.0.0.1:37697/ using document root <none> and password file <none>
I20260927 15:18:20.287374 31202 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:20.287500 31202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:20.287797 31202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:20.291064 31202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "7f80c11a9c0b454ab408abac7685c30b"
format_stamp: "Formatted at 2026-09-27 15:18:20 on dist-test-slave-f5h6"
server_key: "0c16c5da9b4af6ef117098cb2085cc97"
server_key_iv: "0687aa3e95b32b01ff9816bfa6d1ac6a"
server_key_version: "encryptionkey@0"
I20260927 15:18:20.291887 31202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "7f80c11a9c0b454ab408abac7685c30b"
format_stamp: "Formatted at 2026-09-27 15:18:20 on dist-test-slave-f5h6"
server_key: "0c16c5da9b4af6ef117098cb2085cc97"
server_key_iv: "0687aa3e95b32b01ff9816bfa6d1ac6a"
server_key_version: "encryptionkey@0"
I20260927 15:18:20.297559 31202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 15:18:20.301949 31217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:20.303503 31202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:18:20.303700 31202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "7f80c11a9c0b454ab408abac7685c30b"
format_stamp: "Formatted at 2026-09-27 15:18:20 on dist-test-slave-f5h6"
server_key: "0c16c5da9b4af6ef117098cb2085cc97"
server_key_iv: "0687aa3e95b32b01ff9816bfa6d1ac6a"
server_key_version: "encryptionkey@0"
I20260927 15:18:20.303918 31202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:20.331796 31202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:20.336601 31202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:20.336898 31202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:20.355734 31202 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39917
I20260927 15:18:20.355773 31268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39917 every 8 connection(s)
I20260927 15:18:20.357206 31202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:20.361406 31269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:20.365017 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 31202
I20260927 15:18:20.365228 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:20.365777 10373 external_mini_cluster.cc:1428] Setting key 263ceff0b160dcc53b5ab2e10aafe6bd
I20260927 15:18:20.375186 31269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b: Bootstrap starting.
I20260927 15:18:20.379062 31269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:20.380221 31269 log.cc:824] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:20.380553 31272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:20.367959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50441 (local address 127.0.0.1:39917)
0927 15:18:20.368764 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:20.368776 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:20.368819 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:18:20.370604 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:20.370628 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:20.371082 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:20.371430 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:20.372057 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:20.373459 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:20.375544 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:20.376317 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:20.376470 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:20.377749 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:20.377758 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:20.377773 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:20.378379 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:20.378463 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:20.378489 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:20.378583 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:20.378905 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 15:18:20.379214 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":359,"threads_started":1}
I20260927 15:18:20.383455 31269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b: No bootstrap required, opened a new log
I20260927 15:18:20.384552 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:18:20.392141 31269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f80c11a9c0b454ab408abac7685c30b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39917 } }
I20260927 15:18:20.392603 31269 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:20.392704 31269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f80c11a9c0b454ab408abac7685c30b, State: Initialized, Role: FOLLOWER
I20260927 15:18:20.393342 31269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f80c11a9c0b454ab408abac7685c30b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39917 } }
I20260927 15:18:20.393610 31269 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:18:20.393760 31269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:18:20.393975 31269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:20.395737 31272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:20.389300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43659 (local address 127.0.0.1:39917)
0927 15:18:20.389496 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:20.389503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:20.389901 (+ 398us) server_negotiation.cc:407] Connection header received
0927 15:18:20.390116 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:20.390121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:20.390267 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:20.390368 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:20.391017 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:20.391767 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:20.393213 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:20.393692 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:20.393826 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:20.394992 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:20.394997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:20.395003 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:20.395024 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:20.395068 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:20.395086 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:20.395169 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:20.395313 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 15:18:20.395417 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:18:20.396884 31269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f80c11a9c0b454ab408abac7685c30b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39917 } }
I20260927 15:18:20.397512 31269 leader_election.cc:302] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f80c11a9c0b454ab408abac7685c30b; no voters:
I20260927 15:18:20.398099 31269 leader_election.cc:288] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:18:20.398221 31279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:20.398936 31279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b [term 1 LEADER]: Becoming Leader. State: Replica: 7f80c11a9c0b454ab408abac7685c30b, State: Running, Role: LEADER
I20260927 15:18:20.399572 31279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f80c11a9c0b454ab408abac7685c30b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39917 } }
I20260927 15:18:20.400414 31269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:20.402130 31280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f80c11a9c0b454ab408abac7685c30b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f80c11a9c0b454ab408abac7685c30b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39917 } } }
I20260927 15:18:20.402998 31280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:20.402688 31281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f80c11a9c0b454ab408abac7685c30b. Latest consensus state: current_term: 1 leader_uuid: "7f80c11a9c0b454ab408abac7685c30b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f80c11a9c0b454ab408abac7685c30b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39917 } } }
I20260927 15:18:20.403455 31281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:20.404604 31284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:20.410027 31284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:20.419013 31284 catalog_manager.cc:1347] Generated new cluster ID: 2dfd2faff00e4cc994f72c944fb615b2
I20260927 15:18:20.419134 31284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:20.428081 31284 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:20.430003 31284 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:20.438934 31284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f80c11a9c0b454ab408abac7685c30b: Generated new TSK 0
I20260927 15:18:20.440056 31284 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:18:20.447702 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 31202
W20260927 15:18:20.448803 31264 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50441
W20260927 15:18:20.449548 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 31202: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:18:20.449676 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 31202
2026-09-27T15:18:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1826 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:18:20 dist-test-slave-f5h6 krb5kdc[31300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:18:20 dist-test-slave-f5h6 krb5kdc[31300](info): set up 2 sockets
Sep 27 15:18:20 dist-test-slave-f5h6 krb5kdc[31300](info): commencing operation
krb5kdc: starting...
W20260927 15:18:21.486332 10373 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:18:21 dist-test-slave-f5h6 krb5kdc[31300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:18:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:18:21.749032 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45943
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:22.082285 31313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:22.082597 31313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:22.082695 31313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:22.090152 31313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:22.090277 31313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:22.090329 31313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:22.090389 31313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:22.090430 31313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:22.101689 31313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:22.103055 31313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:22.104331 31313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:22.117720 31319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:22.117609 31321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:22.117997 31318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:22.118556 31313 server_base.cc:1034] running on GCE node
I20260927 15:18:22.119413 31313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:22.121050 31313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:22.122326 31313 hybrid_clock.cc:648] HybridClock initialized: now 1790522302122282 us; error 37 us; skew 500 ppm
I20260927 15:18:22.122772 31313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:22 dist-test-slave-f5h6 krb5kdc[31300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:22.126137 31313 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:18:22.127815 31313 webserver.cc:460] Webserver started at http://127.0.0.1:35049/ using document root <none> and password file <none>
I20260927 15:18:22.128563 31313 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:22.128692 31313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:22.129024 31313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:22.132153 31313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "33d2d56da66c451e9f8c6f2912378ea7"
format_stamp: "Formatted at 2026-09-27 15:18:22 on dist-test-slave-f5h6"
server_key: "899e0455649e953738af69190743dc36"
server_key_iv: "26ec51a66c29c3a9bb628cdd8084c4cd"
server_key_version: "encryptionkey@0"
I20260927 15:18:22.132956 31313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "33d2d56da66c451e9f8c6f2912378ea7"
format_stamp: "Formatted at 2026-09-27 15:18:22 on dist-test-slave-f5h6"
server_key: "899e0455649e953738af69190743dc36"
server_key_iv: "26ec51a66c29c3a9bb628cdd8084c4cd"
server_key_version: "encryptionkey@0"
I20260927 15:18:22.138449 31313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 15:18:22.142530 31328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:22.144150 31313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:18:22.144423 31313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "33d2d56da66c451e9f8c6f2912378ea7"
format_stamp: "Formatted at 2026-09-27 15:18:22 on dist-test-slave-f5h6"
server_key: "899e0455649e953738af69190743dc36"
server_key_iv: "26ec51a66c29c3a9bb628cdd8084c4cd"
server_key_version: "encryptionkey@0"
I20260927 15:18:22.144685 31313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:22.151676 31313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:22.156221 31313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:22.156520 31313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:22.174117 31313 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45943
I20260927 15:18:22.174157 31379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45943 every 8 connection(s)
I20260927 15:18:22.175549 31313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:22.179306 31380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:22.183776 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 31313
I20260927 15:18:22.183938 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:22.184401 10373 external_mini_cluster.cc:1428] Setting key a3b42e7f4eb4bf1d128543332d69f61c
I20260927 15:18:22.190181 31380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7: Bootstrap starting.
I20260927 15:18:22.192047 31383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:22.186093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45167 (local address 127.0.0.1:45943)
0927 15:18:22.186877 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:22.186892 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:22.186935 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:18:22.188825 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:22.188848 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:22.189189 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:22.189523 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:22.190533 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:22.190539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:22.190552 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:22.190566 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:22.190631 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:22.190655 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:22.190865 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:22.190991 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 15:18:22.191159 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":332,"threads_started":1}
I20260927 15:18:22.193318 31380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:22.194425 31380 log.cc:824] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:22.195613 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:18:22.197125 31380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7: No bootstrap required, opened a new log
I20260927 15:18:22.202239 31383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:22.199845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51347 (local address 127.0.0.1:45943)
0927 15:18:22.200013 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:22.200023 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:22.200354 (+ 331us) server_negotiation.cc:407] Connection header received
0927 15:18:22.200541 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:22.200545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:22.200596 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:22.200682 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:22.201581 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:22.201585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:22.201588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:22.201603 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:22.201648 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:22.201665 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:22.201765 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:22.201902 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 15:18:22.201967 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:18:22.205837 31380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d2d56da66c451e9f8c6f2912378ea7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45943 } }
I20260927 15:18:22.206390 31380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:22.206528 31380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d2d56da66c451e9f8c6f2912378ea7, State: Initialized, Role: FOLLOWER
I20260927 15:18:22.207175 31380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d2d56da66c451e9f8c6f2912378ea7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45943 } }
I20260927 15:18:22.207403 31380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:18:22.207542 31380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:18:22.207710 31380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:22.210049 31380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d2d56da66c451e9f8c6f2912378ea7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45943 } }
I20260927 15:18:22.210582 31380 leader_election.cc:302] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33d2d56da66c451e9f8c6f2912378ea7; no voters:
I20260927 15:18:22.211138 31380 leader_election.cc:288] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:18:22.211270 31390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:22.211905 31390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7 [term 1 LEADER]: Becoming Leader. State: Replica: 33d2d56da66c451e9f8c6f2912378ea7, State: Running, Role: LEADER
I20260927 15:18:22.212428 31390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d2d56da66c451e9f8c6f2912378ea7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45943 } }
I20260927 15:18:22.213132 31380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:22.215111 31391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33d2d56da66c451e9f8c6f2912378ea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d2d56da66c451e9f8c6f2912378ea7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45943 } } }
I20260927 15:18:22.215548 31391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:22.216039 31392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33d2d56da66c451e9f8c6f2912378ea7. Latest consensus state: current_term: 1 leader_uuid: "33d2d56da66c451e9f8c6f2912378ea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d2d56da66c451e9f8c6f2912378ea7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45943 } } }
I20260927 15:18:22.216313 31392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:22.219347 31397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:22.224687 31397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:22.234122 31397 catalog_manager.cc:1347] Generated new cluster ID: b156fa0e95ac480d89b5457e6bbdae36
I20260927 15:18:22.234249 31397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:22.247462 31397 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:22.247953 31397 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:22.256804 31397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33d2d56da66c451e9f8c6f2912378ea7: Generated new TSK 0
I20260927 15:18:22.257781 31397 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:18:22.268514 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 31313
W20260927 15:18:22.269721 31375 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45167
W20260927 15:18:22.270481 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 31313: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:18:22.270639 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 31313
2026-09-27T15:18:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1820 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:18:22 dist-test-slave-f5h6 krb5kdc[31411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:18:22 dist-test-slave-f5h6 krb5kdc[31411](info): set up 2 sockets
Sep 27 15:18:22 dist-test-slave-f5h6 krb5kdc[31411](info): commencing operation
krb5kdc: starting...
W20260927 15:18:23.290405 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:18:23 dist-test-slave-f5h6 krb5kdc[31411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:18:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:18:23.547159 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43239
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:23.881349 31424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:23.881671 31424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:23.881815 31424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:23.889175 31424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:23.889297 31424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:23.889366 31424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:23.889451 31424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:23.889506 31424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:23.899878 31424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:23.901221 31424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:23.902614 31424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:23.914470 31430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:23.914525 31429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:23.914549 31432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:23.915865 31424 server_base.cc:1034] running on GCE node
I20260927 15:18:23.916682 31424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:23.918505 31424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:23.919739 31424 hybrid_clock.cc:648] HybridClock initialized: now 1790522303919688 us; error 32 us; skew 500 ppm
I20260927 15:18:23.920125 31424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:23 dist-test-slave-f5h6 krb5kdc[31411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:23.923504 31424 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:18:23.925230 31424 webserver.cc:460] Webserver started at http://127.0.0.1:34653/ using document root <none> and password file <none>
I20260927 15:18:23.925987 31424 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:23.926110 31424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:23.926415 31424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:23.929584 31424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "e0fa1ff45d64460e83aeba9986e1f930"
format_stamp: "Formatted at 2026-09-27 15:18:23 on dist-test-slave-f5h6"
server_key: "8f0cd7c9eae8489d20a4f2061a282132"
server_key_iv: "e2ccbb8d2e8c7e7dfc2fd53105a6fbfe"
server_key_version: "encryptionkey@0"
I20260927 15:18:23.930462 31424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "e0fa1ff45d64460e83aeba9986e1f930"
format_stamp: "Formatted at 2026-09-27 15:18:23 on dist-test-slave-f5h6"
server_key: "8f0cd7c9eae8489d20a4f2061a282132"
server_key_iv: "e2ccbb8d2e8c7e7dfc2fd53105a6fbfe"
server_key_version: "encryptionkey@0"
I20260927 15:18:23.935966 31424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:18:23.939644 31439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:23.941035 31424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:18:23.941231 31424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "e0fa1ff45d64460e83aeba9986e1f930"
format_stamp: "Formatted at 2026-09-27 15:18:23 on dist-test-slave-f5h6"
server_key: "8f0cd7c9eae8489d20a4f2061a282132"
server_key_iv: "e2ccbb8d2e8c7e7dfc2fd53105a6fbfe"
server_key_version: "encryptionkey@0"
I20260927 15:18:23.941475 31424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:23.995550 31424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:23.999940 31424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:24.000197 31424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:24.017207 31424 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43239
I20260927 15:18:24.017241 31490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43239 every 8 connection(s)
I20260927 15:18:24.018616 31424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:24.022385 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 31424
I20260927 15:18:24.022544 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:24.022596 31491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:24.022986 10373 external_mini_cluster.cc:1428] Setting key a526fde3c0c262b70a8ed82c30020b18
I20260927 15:18:24.035333 31491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930: Bootstrap starting.
Sep 27 15:18:24 dist-test-slave-f5h6 krb5kdc[31411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:18:24.039016 31491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:24.040146 31491 log.cc:824] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:24.043197 31491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930: No bootstrap required, opened a new log
I20260927 15:18:24.046276 31494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:24.024793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:59723 (local address 127.0.0.1:43239)
0927 15:18:24.025634 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:24.025645 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:24.025692 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:18:24.027498 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:24.027521 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:24.027936 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:24.028318 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:24.029751 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:24.031091 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:24.032334 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:24.033075 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:24.033251 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:24.037737 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:24.037747 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:24.037765 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:24.037810 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:24.041120 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:24.042118 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:24.042127 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:24.042134 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:24.042333 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:24.042816 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:24.042820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:24.042822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:24.043424 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:24.043739 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:24.044407 (+ 668us) server_negotiation.cc:299] Negotiation successful
0927 15:18:24.045033 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":349,"threads_started":1}
I20260927 15:18:24.050901 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:18:24.053550 31491 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fa1ff45d64460e83aeba9986e1f930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43239 } }
I20260927 15:18:24.054122 31491 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:24.054246 31491 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0fa1ff45d64460e83aeba9986e1f930, State: Initialized, Role: FOLLOWER
I20260927 15:18:24.055054 31491 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fa1ff45d64460e83aeba9986e1f930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43239 } }
I20260927 15:18:24.055368 31491 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:18:24.055518 31491 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:18:24.055693 31491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:24.058481 31491 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fa1ff45d64460e83aeba9986e1f930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43239 } }
I20260927 15:18:24.058980 31491 leader_election.cc:302] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0fa1ff45d64460e83aeba9986e1f930; no voters:
I20260927 15:18:24.059608 31491 leader_election.cc:288] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:18:24.059707 31501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:24.061461 31501 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930 [term 1 LEADER]: Becoming Leader. State: Replica: e0fa1ff45d64460e83aeba9986e1f930, State: Running, Role: LEADER
I20260927 15:18:24.062156 31491 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:24.062150 31501 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fa1ff45d64460e83aeba9986e1f930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43239 } }
I20260927 15:18:24.065363 31503 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0fa1ff45d64460e83aeba9986e1f930. Latest consensus state: current_term: 1 leader_uuid: "e0fa1ff45d64460e83aeba9986e1f930" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fa1ff45d64460e83aeba9986e1f930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43239 } } }
I20260927 15:18:24.066368 31503 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:24.066031 31502 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0fa1ff45d64460e83aeba9986e1f930" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fa1ff45d64460e83aeba9986e1f930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43239 } } }
I20260927 15:18:24.066915 31502 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:24.073468 31508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:24.075383 31494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:24.056336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:35957 (local address 127.0.0.1:43239)
0927 15:18:24.056541 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:24.056548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:24.059150 (+ 2602us) server_negotiation.cc:407] Connection header received
0927 15:18:24.059321 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:24.059326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:24.059406 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:24.059489 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:24.061256 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:24.062292 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:24.063604 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:24.064377 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:24.064505 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:24.068119 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:24.068127 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:24.068133 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:24.068181 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:24.071611 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:24.072653 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:24.072658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:24.072661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:24.072799 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:24.073297 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:24.073303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:24.073306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:24.073647 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:24.073929 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:24.074788 (+ 859us) server_negotiation.cc:299] Negotiation successful
0927 15:18:24.075004 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:18:24.079550 31508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:24.089088 31508 catalog_manager.cc:1347] Generated new cluster ID: 71eedfd0e105465fb5f001bf2e3a6d55
I20260927 15:18:24.089205 31508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:24.105765 31508 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:24.108600 31508 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:24.124253 31508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0fa1ff45d64460e83aeba9986e1f930: Generated new TSK 0
I20260927 15:18:24.125242 31508 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:18:24.134102 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 31424
I20260927 15:18:24.135653 31486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:24.228615 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 31424
W20260927 15:18:24.243404 31409 connection.cc:535] client connection to 127.0.0.1:43239 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43239): Connection reset by peer (error 104)
2026-09-27T15:18:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1958 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:18:24 dist-test-slave-f5h6 krb5kdc[31523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:18:24 dist-test-slave-f5h6 krb5kdc[31523](info): set up 2 sockets
Sep 27 15:18:24 dist-test-slave-f5h6 krb5kdc[31523](info): commencing operation
krb5kdc: starting...
W20260927 15:18:25.250274 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:18:25 dist-test-slave-f5h6 krb5kdc[31523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:18:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:18:25.502027 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:25.831384 31536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:25.831666 31536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:25.831782 31536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:25.839046 31536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:25.839165 31536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:25.839212 31536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:25.839293 31536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:25.839335 31536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:25.849392 31536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:25.850728 31536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:25.852032 31536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:25.864593 31541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:25.864629 31542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:25.864655 31544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:25.864981 31536 server_base.cc:1034] running on GCE node
I20260927 15:18:25.866096 31536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:25.867591 31536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:25.868873 31536 hybrid_clock.cc:648] HybridClock initialized: now 1790522305868821 us; error 36 us; skew 500 ppm
I20260927 15:18:25.869277 31536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:25 dist-test-slave-f5h6 krb5kdc[31523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:25.872587 31536 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:18:25.874308 31536 webserver.cc:460] Webserver started at http://127.0.0.1:39077/ using document root <none> and password file <none>
I20260927 15:18:25.875033 31536 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:25.875161 31536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:25.875476 31536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:25.878753 31536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "bb6e31674f5b4a73a72fa95058a20992"
format_stamp: "Formatted at 2026-09-27 15:18:25 on dist-test-slave-f5h6"
server_key: "6cca90998cfab074138296bc1038f2ef"
server_key_iv: "fdeb50160ce9d2d4d237dd8314f7678e"
server_key_version: "encryptionkey@0"
I20260927 15:18:25.879582 31536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "bb6e31674f5b4a73a72fa95058a20992"
format_stamp: "Formatted at 2026-09-27 15:18:25 on dist-test-slave-f5h6"
server_key: "6cca90998cfab074138296bc1038f2ef"
server_key_iv: "fdeb50160ce9d2d4d237dd8314f7678e"
server_key_version: "encryptionkey@0"
I20260927 15:18:25.885161 31536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:18:25.888854 31551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:25.890308 31536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:18:25.890517 31536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "bb6e31674f5b4a73a72fa95058a20992"
format_stamp: "Formatted at 2026-09-27 15:18:25 on dist-test-slave-f5h6"
server_key: "6cca90998cfab074138296bc1038f2ef"
server_key_iv: "fdeb50160ce9d2d4d237dd8314f7678e"
server_key_version: "encryptionkey@0"
I20260927 15:18:25.890713 31536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:25.907752 31536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:25.912814 31536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:25.913103 31536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:25.931281 31536 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39371
I20260927 15:18:25.931303 31602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39371 every 8 connection(s)
I20260927 15:18:25.932631 31536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:25.936610 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 31536
I20260927 15:18:25.936736 31603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:25.936813 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:25.937381 10373 external_mini_cluster.cc:1428] Setting key 46e0bab3a6d09a5e39a8bc963a12d8c5
I20260927 15:18:25.952255 31606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:25.939270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:48297 (local address 127.0.0.1:39371)
0927 15:18:25.940073 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:25.940089 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:25.940135 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:18:25.942017 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:25.942045 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:25.942528 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:25.942923 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:25.943652 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:25.945061 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:25.947015 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:25.947764 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:25.947936 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:25.949339 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:25.949354 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:25.949375 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:25.950168 (+ 793us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:25.950283 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:25.950317 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:25.950470 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:25.950994 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 15:18:25.951197 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":348,"threads_started":1}
I20260927 15:18:25.952415 31603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992: Bootstrap starting.
I20260927 15:18:25.956013 31603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:25.956367 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:18:25.957322 31603 log.cc:824] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:25.960942 31603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992: No bootstrap required, opened a new log
I20260927 15:18:25.968315 31606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:25.961369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:43235 (local address 127.0.0.1:39371)
0927 15:18:25.961581 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:25.961587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:25.962221 (+ 634us) server_negotiation.cc:407] Connection header received
0927 15:18:25.962416 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:25.962422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:25.962507 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:25.962594 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:25.963164 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:25.964037 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:25.965395 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:25.966013 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:25.966141 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:25.967440 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:25.967455 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:25.967460 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:25.967485 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:25.967532 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:25.967560 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:25.967700 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:25.967945 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 15:18:25.968019 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:18:25.969887 31603 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6e31674f5b4a73a72fa95058a20992" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39371 } }
I20260927 15:18:25.970299 31603 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:25.970402 31603 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6e31674f5b4a73a72fa95058a20992, State: Initialized, Role: FOLLOWER
I20260927 15:18:25.971124 31603 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6e31674f5b4a73a72fa95058a20992" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39371 } }
I20260927 15:18:25.971381 31603 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:18:25.971580 31603 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:18:25.971740 31603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:25.974162 31603 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6e31674f5b4a73a72fa95058a20992" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39371 } }
I20260927 15:18:25.974747 31603 leader_election.cc:302] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb6e31674f5b4a73a72fa95058a20992; no voters:
I20260927 15:18:25.975512 31603 leader_election.cc:288] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:18:25.975600 31613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:25.976303 31613 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992 [term 1 LEADER]: Becoming Leader. State: Replica: bb6e31674f5b4a73a72fa95058a20992, State: Running, Role: LEADER
I20260927 15:18:25.976859 31613 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6e31674f5b4a73a72fa95058a20992" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39371 } }
I20260927 15:18:25.977694 31603 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:25.980118 31614 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb6e31674f5b4a73a72fa95058a20992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6e31674f5b4a73a72fa95058a20992" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39371 } } }
I20260927 15:18:25.980517 31614 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:25.981140 31615 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb6e31674f5b4a73a72fa95058a20992. Latest consensus state: current_term: 1 leader_uuid: "bb6e31674f5b4a73a72fa95058a20992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6e31674f5b4a73a72fa95058a20992" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39371 } } }
I20260927 15:18:25.981418 31615 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:25.982893 31621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:25.987540 31621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:25.996973 31621 catalog_manager.cc:1347] Generated new cluster ID: eb1d73bc328149eabec11e5df1a2b153
I20260927 15:18:25.997098 31621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:26.008620 31621 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:26.010535 31621 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:26.032482 31621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb6e31674f5b4a73a72fa95058a20992: Generated new TSK 0
I20260927 15:18:26.033726 31621 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:18:26.041417 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 31536
W20260927 15:18:26.043121 31598 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.232:48297
W20260927 15:18:26.043803 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 31536: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:18:26.043965 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 31536
W20260927 15:18:26.058154 31521 connection.cc:535] client connection to 127.0.0.1:39371 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39371): Connection reset by peer (error 104)
2026-09-27T15:18:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1814 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:18:26 dist-test-slave-f5h6 krb5kdc[31634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:18:26 dist-test-slave-f5h6 krb5kdc[31634](info): set up 2 sockets
Sep 27 15:18:26 dist-test-slave-f5h6 krb5kdc[31634](info): commencing operation
krb5kdc: starting...
W20260927 15:18:27.078138 10373 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:18:27 dist-test-slave-f5h6 krb5kdc[31634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:18:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:18:27.341885 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44843
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:27.673846 31647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:27.674242 31647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:27.674362 31647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:27.682017 31647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:27.682149 31647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:27.682200 31647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:27.682284 31647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:27.682348 31647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:27.692674 31647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:27.694119 31647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:27.695577 31647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:27.707674 31652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:27.707695 31653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:27.707756 31655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:27.708417 31647 server_base.cc:1034] running on GCE node
I20260927 15:18:27.709193 31647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:27.710698 31647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:27.711967 31647 hybrid_clock.cc:648] HybridClock initialized: now 1790522307711903 us; error 53 us; skew 500 ppm
I20260927 15:18:27.712329 31647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:27 dist-test-slave-f5h6 krb5kdc[31634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:27.715469 31647 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:18:27.716925 31647 webserver.cc:460] Webserver started at http://127.0.0.1:38691/ using document root <none> and password file <none>
I20260927 15:18:27.717564 31647 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:27.717660 31647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:27.717967 31647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:27.720923 31647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "501bde4db67d4e179713a5fd45f8fe41"
format_stamp: "Formatted at 2026-09-27 15:18:27 on dist-test-slave-f5h6"
server_key: "5990a13770a91fa7cde5f8b70bf5b904"
server_key_iv: "fe74d4e2cd09aa3c2df434cd1d96ecd7"
server_key_version: "encryptionkey@0"
I20260927 15:18:27.721647 31647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "501bde4db67d4e179713a5fd45f8fe41"
format_stamp: "Formatted at 2026-09-27 15:18:27 on dist-test-slave-f5h6"
server_key: "5990a13770a91fa7cde5f8b70bf5b904"
server_key_iv: "fe74d4e2cd09aa3c2df434cd1d96ecd7"
server_key_version: "encryptionkey@0"
I20260927 15:18:27.727066 31647 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 15:18:27.730844 31662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:27.732363 31647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:18:27.732513 31647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "501bde4db67d4e179713a5fd45f8fe41"
format_stamp: "Formatted at 2026-09-27 15:18:27 on dist-test-slave-f5h6"
server_key: "5990a13770a91fa7cde5f8b70bf5b904"
server_key_iv: "fe74d4e2cd09aa3c2df434cd1d96ecd7"
server_key_version: "encryptionkey@0"
I20260927 15:18:27.732692 31647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:27.738226 31647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:27.742314 31647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:27.742566 31647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:27.760623 31647 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44843
I20260927 15:18:27.760690 31713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44843 every 8 connection(s)
I20260927 15:18:27.762013 31647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:27.765884 31714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:27.766922 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 31647
I20260927 15:18:27.767063 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:27.767516 10373 external_mini_cluster.cc:1428] Setting key 73ba8b1d5a83358de7cfd29d21df932e
I20260927 15:18:27.776012 31717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:27.769517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:57815 (local address 127.0.0.1:44843)
0927 15:18:27.770439 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:27.770451 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:27.770495 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:18:27.772483 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:27.772510 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:27.772908 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:27.773324 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:27.774409 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:27.774416 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:27.774429 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:27.774446 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:27.774513 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:27.774533 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:27.774740 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:27.774867 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 15:18:27.775062 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":425,"threads_started":1}
I20260927 15:18:27.779910 31714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41: Bootstrap starting.
I20260927 15:18:27.780324 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:18:27.784029 31714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:27.785429 31714 log.cc:824] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:27.788383 31717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:27.786006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:43849 (local address 127.0.0.1:44843)
0927 15:18:27.786208 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:27.786214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:27.786409 (+ 195us) server_negotiation.cc:407] Connection header received
0927 15:18:27.786586 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:27.786592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:27.786656 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:27.786786 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:27.787775 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:27.787780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:27.787783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:27.787797 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:27.787836 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:27.787850 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:27.787944 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:27.788070 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 15:18:27.788120 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:18:27.789083 31714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41: No bootstrap required, opened a new log
I20260927 15:18:27.796509 31714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501bde4db67d4e179713a5fd45f8fe41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44843 } }
I20260927 15:18:27.796880 31714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:27.796954 31714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501bde4db67d4e179713a5fd45f8fe41, State: Initialized, Role: FOLLOWER
I20260927 15:18:27.797449 31714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501bde4db67d4e179713a5fd45f8fe41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44843 } }
I20260927 15:18:27.797655 31714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:18:27.797753 31714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:18:27.797914 31714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:27.800077 31714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501bde4db67d4e179713a5fd45f8fe41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44843 } }
I20260927 15:18:27.800535 31714 leader_election.cc:302] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 501bde4db67d4e179713a5fd45f8fe41; no voters:
I20260927 15:18:27.801064 31714 leader_election.cc:288] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:18:27.801199 31724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:27.802279 31724 raft_consensus.cc:686] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41 [term 1 LEADER]: Becoming Leader. State: Replica: 501bde4db67d4e179713a5fd45f8fe41, State: Running, Role: LEADER
I20260927 15:18:27.802903 31724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501bde4db67d4e179713a5fd45f8fe41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44843 } }
I20260927 15:18:27.803323 31714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:27.805505 31725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "501bde4db67d4e179713a5fd45f8fe41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501bde4db67d4e179713a5fd45f8fe41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44843 } } }
I20260927 15:18:27.805848 31725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:27.806718 31731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:27.806547 31726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 501bde4db67d4e179713a5fd45f8fe41. Latest consensus state: current_term: 1 leader_uuid: "501bde4db67d4e179713a5fd45f8fe41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501bde4db67d4e179713a5fd45f8fe41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44843 } } }
I20260927 15:18:27.806890 31726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:27.812523 31731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:27.823238 31731 catalog_manager.cc:1347] Generated new cluster ID: 3d16da6d99b44df9a5e8995351117104
I20260927 15:18:27.823392 31731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:27.846547 31731 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:27.847043 31731 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:27.858562 31731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 501bde4db67d4e179713a5fd45f8fe41: Generated new TSK 0
I20260927 15:18:27.859508 31731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:18:27.873214 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 31647
W20260927 15:18:27.874322 31709 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.232:57815
W20260927 15:18:27.875057 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 31647: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:18:27.875190 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 31647
2026-09-27T15:18:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1830 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:18:27 dist-test-slave-f5h6 krb5kdc[31745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:18:27 dist-test-slave-f5h6 krb5kdc[31745](info): set up 2 sockets
Sep 27 15:18:27 dist-test-slave-f5h6 krb5kdc[31745](info): commencing operation
krb5kdc: starting...
W20260927 15:18:28.902173 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:18:29 dist-test-slave-f5h6 krb5kdc[31745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:18:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:18:29.161827 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35495
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:29.490707 31758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:29.491031 31758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:29.491156 31758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:29.498736 31758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:29.498865 31758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:29.498926 31758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:29.499017 31758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:29.499078 31758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:29.509284 31758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35495
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:29.510854 31758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:29.512352 31758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:29.524875 31763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:29.524905 31766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:29.524842 31764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:29.525636 31758 server_base.cc:1034] running on GCE node
I20260927 15:18:29.526500 31758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:29.528038 31758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:29.529278 31758 hybrid_clock.cc:648] HybridClock initialized: now 1790522309529221 us; error 48 us; skew 500 ppm
I20260927 15:18:29.529676 31758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:29 dist-test-slave-f5h6 krb5kdc[31745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:29.532841 31758 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:18:29.534343 31758 webserver.cc:460] Webserver started at http://127.0.0.1:40957/ using document root <none> and password file <none>
I20260927 15:18:29.535003 31758 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:29.535105 31758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:29.535370 31758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:29.538331 31758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "acfd1c2bd6b044af9f9f3e0233927ddb"
format_stamp: "Formatted at 2026-09-27 15:18:29 on dist-test-slave-f5h6"
server_key: "73bfab77bf883127862415fcd1258815"
server_key_iv: "e644ddbd8f491da04ffc0aae259d3d17"
server_key_version: "encryptionkey@0"
I20260927 15:18:29.539055 31758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "acfd1c2bd6b044af9f9f3e0233927ddb"
format_stamp: "Formatted at 2026-09-27 15:18:29 on dist-test-slave-f5h6"
server_key: "73bfab77bf883127862415fcd1258815"
server_key_iv: "e644ddbd8f491da04ffc0aae259d3d17"
server_key_version: "encryptionkey@0"
I20260927 15:18:29.544346 31758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 15:18:29.548321 31773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:29.549795 31758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:18:29.550040 31758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "acfd1c2bd6b044af9f9f3e0233927ddb"
format_stamp: "Formatted at 2026-09-27 15:18:29 on dist-test-slave-f5h6"
server_key: "73bfab77bf883127862415fcd1258815"
server_key_iv: "e644ddbd8f491da04ffc0aae259d3d17"
server_key_version: "encryptionkey@0"
I20260927 15:18:29.550244 31758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:29.571240 31758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:29.575737 31758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:29.575997 31758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:29.595474 31758 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35495
I20260927 15:18:29.595557 31824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35495 every 8 connection(s)
I20260927 15:18:29.596916 31758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:29.600682 31825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:29.606956 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 31758
I20260927 15:18:29.607120 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:29.607527 10373 external_mini_cluster.cc:1428] Setting key 5995815d95a21b0dac0e3fd6fb0fa23f
I20260927 15:18:29.611609 31825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb: Bootstrap starting.
I20260927 15:18:29.615052 31825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:29.616351 31825 log.cc:824] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:29.619648 31825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb: No bootstrap required, opened a new log
Sep 27 15:18:29 dist-test-slave-f5h6 krb5kdc[31745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:18:29.628084 31825 raft_consensus.cc:348] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfd1c2bd6b044af9f9f3e0233927ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35495 } }
I20260927 15:18:29.628480 31825 raft_consensus.cc:374] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:29.628582 31825 raft_consensus.cc:729] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acfd1c2bd6b044af9f9f3e0233927ddb, State: Initialized, Role: FOLLOWER
I20260927 15:18:29.629204 31825 consensus_queue.cc:260] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfd1c2bd6b044af9f9f3e0233927ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35495 } }
I20260927 15:18:29.629448 31825 raft_consensus.cc:388] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:18:29.629601 31825 raft_consensus.cc:482] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:18:29.629818 31825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:29.631031 31828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:29.609312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:53923 (local address 127.0.0.1:35495)
0927 15:18:29.610164 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:29.610176 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:29.610215 (+ 39us) server_negotiation.cc:407] Connection header received
0927 15:18:29.611927 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:29.611951 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:29.612366 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:29.612739 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:29.614474 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:29.616142 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:29.617805 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:29.618542 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:29.618713 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:29.622820 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:29.622828 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:29.622845 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:29.622888 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:29.626438 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:29.627254 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:29.627265 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:29.627272 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:29.627448 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:29.628055 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:29.628059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:29.628061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:29.628698 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:29.629113 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:29.629549 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 15:18:29.629931 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":323,"threads_started":1}
I20260927 15:18:29.632701 31825 raft_consensus.cc:504] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfd1c2bd6b044af9f9f3e0233927ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35495 } }
I20260927 15:18:29.633174 31825 leader_election.cc:302] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acfd1c2bd6b044af9f9f3e0233927ddb; no voters:
I20260927 15:18:29.634060 31825 leader_election.cc:288] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:18:29.634207 31830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:29.635052 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:18:29.635941 31825 sys_catalog.cc:564] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:29.638375 31830 raft_consensus.cc:686] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb [term 1 LEADER]: Becoming Leader. State: Replica: acfd1c2bd6b044af9f9f3e0233927ddb, State: Running, Role: LEADER
I20260927 15:18:29.639038 31830 consensus_queue.cc:237] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfd1c2bd6b044af9f9f3e0233927ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35495 } }
I20260927 15:18:29.642033 31832 sys_catalog.cc:455] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb [sys.catalog]: SysCatalogTable state changed. Reason: New leader acfd1c2bd6b044af9f9f3e0233927ddb. Latest consensus state: current_term: 1 leader_uuid: "acfd1c2bd6b044af9f9f3e0233927ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfd1c2bd6b044af9f9f3e0233927ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35495 } } }
I20260927 15:18:29.642494 31832 sys_catalog.cc:458] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:29.643060 31831 sys_catalog.cc:455] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acfd1c2bd6b044af9f9f3e0233927ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfd1c2bd6b044af9f9f3e0233927ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35495 } } }
I20260927 15:18:29.643334 31831 sys_catalog.cc:458] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:29.647199 31838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:29.654896 31838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:29.667018 31838 catalog_manager.cc:1347] Generated new cluster ID: 021f377505fe439ab09c835afd7cdeac
I20260927 15:18:29.667232 31838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:29.683986 31828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:29.650382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:39367 (local address 127.0.0.1:35495)
0927 15:18:29.650583 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:29.650591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:29.664351 (+ 13760us) server_negotiation.cc:407] Connection header received
0927 15:18:29.664674 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:29.664679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:29.664773 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:29.664860 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:29.667215 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:29.668530 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:29.670658 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:29.671375 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:29.671476 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:29.675150 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:29.675158 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:29.675164 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:29.675220 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:29.679201 (+ 3981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:29.680399 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:29.680406 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:29.680410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:29.680616 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:29.681174 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:29.681179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:29.681183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:29.681670 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:29.681973 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:29.683326 (+ 1353us) server_negotiation.cc:299] Negotiation successful
0927 15:18:29.683552 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 15:18:29.700001 31838 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:29.701746 31838 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:29.720738 31838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acfd1c2bd6b044af9f9f3e0233927ddb: Generated new TSK 0
I20260927 15:18:29.721880 31838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:18:29.735514 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 31758
I20260927 15:18:29.736810 31820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:29.832597 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 31758
W20260927 15:18:29.847077 31743 connection.cc:535] client connection to 127.0.0.1:35495 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35495): Connection reset by peer (error 104)
2026-09-27T15:18:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1957 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:18:29 dist-test-slave-f5h6 krb5kdc[31857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:18:29 dist-test-slave-f5h6 krb5kdc[31857](info): set up 2 sockets
Sep 27 15:18:29 dist-test-slave-f5h6 krb5kdc[31857](info): commencing operation
krb5kdc: starting...
W20260927 15:18:30.870137 10373 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.006s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:18:31 dist-test-slave-f5h6 krb5kdc[31857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:18:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:18:31.122778 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33499
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:31.471489 31870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:31.471787 31870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:31.471882 31870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:31.479240 31870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:31.479355 31870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:31.479408 31870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:31.479458 31870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:31.479501 31870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:31.489931 31870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33499
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:31.491226 31870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:31.492491 31870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:31.504168 31878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:31.504240 31875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:31.504266 31876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:31.505152 31870 server_base.cc:1034] running on GCE node
I20260927 15:18:31.506086 31870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:31.507628 31870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:31.508898 31870 hybrid_clock.cc:648] HybridClock initialized: now 1790522311508846 us; error 41 us; skew 500 ppm
I20260927 15:18:31.509274 31870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:31 dist-test-slave-f5h6 krb5kdc[31857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:31.512377 31870 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:18:31.514096 31870 webserver.cc:460] Webserver started at http://127.0.0.1:33587/ using document root <none> and password file <none>
I20260927 15:18:31.514779 31870 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:31.514899 31870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:31.515204 31870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:31.518270 31870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "94d10859447044f7bfe18c1406713f9e"
format_stamp: "Formatted at 2026-09-27 15:18:31 on dist-test-slave-f5h6"
server_key: "fa4946ea38505a39d8cc0abf98d9737f"
server_key_iv: "92fa38ffa354c82da68a09614e36d777"
server_key_version: "encryptionkey@0"
I20260927 15:18:31.519014 31870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "94d10859447044f7bfe18c1406713f9e"
format_stamp: "Formatted at 2026-09-27 15:18:31 on dist-test-slave-f5h6"
server_key: "fa4946ea38505a39d8cc0abf98d9737f"
server_key_iv: "92fa38ffa354c82da68a09614e36d777"
server_key_version: "encryptionkey@0"
I20260927 15:18:31.524048 31870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:18:31.527529 31885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:31.528892 31870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:18:31.529042 31870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "94d10859447044f7bfe18c1406713f9e"
format_stamp: "Formatted at 2026-09-27 15:18:31 on dist-test-slave-f5h6"
server_key: "fa4946ea38505a39d8cc0abf98d9737f"
server_key_iv: "92fa38ffa354c82da68a09614e36d777"
server_key_version: "encryptionkey@0"
I20260927 15:18:31.529237 31870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:31.541958 31870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:31.546235 31870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:31.546506 31870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:31.564210 31870 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33499
I20260927 15:18:31.564244 31936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33499 every 8 connection(s)
I20260927 15:18:31.565541 31870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:31.569242 31937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:31.569229 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 31870
I20260927 15:18:31.569428 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:31.569880 10373 external_mini_cluster.cc:1428] Setting key d0636cc0127a7013f2e62095b2f35955
I20260927 15:18:31.584203 31940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:31.571646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:57171 (local address 127.0.0.1:33499)
0927 15:18:31.572476 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:31.572491 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:31.572540 (+ 49us) server_negotiation.cc:407] Connection header received
0927 15:18:31.574153 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:31.574179 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:31.574680 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:31.575093 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:31.575615 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:31.577567 (+ 1952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:31.578977 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:31.579981 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:31.580167 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:31.581279 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:31.581289 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:31.581312 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:31.582075 (+ 763us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:31.582175 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:31.582206 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:31.582351 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:31.582593 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 15:18:31.582770 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":307,"threads_started":1}
I20260927 15:18:31.584512 31937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e: Bootstrap starting.
I20260927 15:18:31.587668 31937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:31.588155 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:18:31.588760 31937 log.cc:824] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:31.591502 31937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e: No bootstrap required, opened a new log
I20260927 15:18:31.599470 31940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:31.592288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:39739 (local address 127.0.0.1:33499)
0927 15:18:31.592481 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:31.592488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:31.592953 (+ 465us) server_negotiation.cc:407] Connection header received
0927 15:18:31.593165 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:31.593169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:31.593234 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:31.593326 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:31.593925 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:31.594760 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:31.596712 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:31.597199 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:31.597301 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:31.598640 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:31.598645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:31.598648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:31.598669 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:31.598708 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:31.598723 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:31.598811 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:31.599045 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 15:18:31.599098 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:18:31.601572 31937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d10859447044f7bfe18c1406713f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33499 } }
I20260927 15:18:31.602020 31937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:31.602119 31937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d10859447044f7bfe18c1406713f9e, State: Initialized, Role: FOLLOWER
I20260927 15:18:31.602819 31937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d10859447044f7bfe18c1406713f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33499 } }
I20260927 15:18:31.603148 31937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:18:31.603302 31937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:18:31.603482 31937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:31.605824 31937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d10859447044f7bfe18c1406713f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33499 } }
I20260927 15:18:31.606433 31937 leader_election.cc:302] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94d10859447044f7bfe18c1406713f9e; no voters:
I20260927 15:18:31.607098 31937 leader_election.cc:288] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:18:31.607236 31947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:31.608666 31947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e [term 1 LEADER]: Becoming Leader. State: Replica: 94d10859447044f7bfe18c1406713f9e, State: Running, Role: LEADER
I20260927 15:18:31.609350 31947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d10859447044f7bfe18c1406713f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33499 } }
I20260927 15:18:31.609670 31937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:31.612149 31949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94d10859447044f7bfe18c1406713f9e. Latest consensus state: current_term: 1 leader_uuid: "94d10859447044f7bfe18c1406713f9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d10859447044f7bfe18c1406713f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33499 } } }
I20260927 15:18:31.612664 31949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:31.612702 31948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94d10859447044f7bfe18c1406713f9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d10859447044f7bfe18c1406713f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33499 } } }
I20260927 15:18:31.613130 31948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:31.615828 31956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:31.622798 31956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:31.631724 31956 catalog_manager.cc:1347] Generated new cluster ID: b2588065347344538a57dac4b7f38e28
I20260927 15:18:31.631879 31956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:31.648052 31956 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:31.649713 31956 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:31.667481 31956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94d10859447044f7bfe18c1406713f9e: Generated new TSK 0
I20260927 15:18:31.668490 31956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:18:31.677469 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 31870
W20260927 15:18:31.679347 31932 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.232:57171
W20260927 15:18:31.680275 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 31870: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:18:31.680444 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 31870
W20260927 15:18:31.695791 31855 connection.cc:535] client connection to 127.0.0.1:33499 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33499): Connection reset by peer (error 104)
2026-09-27T15:18:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1848 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:18:31 dist-test-slave-f5h6 krb5kdc[31968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:18:31 dist-test-slave-f5h6 krb5kdc[31968](info): set up 2 sockets
Sep 27 15:18:31 dist-test-slave-f5h6 krb5kdc[31968](info): commencing operation
krb5kdc: starting...
W20260927 15:18:32.706264 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:18:32 dist-test-slave-f5h6 krb5kdc[31968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:18:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:18:32.967144 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40273
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:33.305120 31981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:33.305471 31981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:33.305610 31981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:33.313187 31981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:33.313338 31981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:33.313414 31981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:33.313493 31981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:33.313551 31981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:33.324338 31981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40273
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:33.325737 31981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:33.327116 31981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:33.339094 31989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:33.339161 31986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:33.339171 31987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:33.340163 31981 server_base.cc:1034] running on GCE node
I20260927 15:18:33.341037 31981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:33.342690 31981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:33.343927 31981 hybrid_clock.cc:648] HybridClock initialized: now 1790522313343855 us; error 67 us; skew 500 ppm
I20260927 15:18:33.344308 31981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:33 dist-test-slave-f5h6 krb5kdc[31968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:33.347506 31981 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:18:33.349157 31981 webserver.cc:460] Webserver started at http://127.0.0.1:37833/ using document root <none> and password file <none>
I20260927 15:18:33.349946 31981 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:33.350075 31981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:33.350374 31981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:33.353650 31981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "7dfd97fad21b49099ad312a68168bcef"
format_stamp: "Formatted at 2026-09-27 15:18:33 on dist-test-slave-f5h6"
server_key: "86520a397a114f67390a07f9044c6ff5"
server_key_iv: "685b898ec46a6bd90790276f3cb8713a"
server_key_version: "encryptionkey@0"
I20260927 15:18:33.354547 31981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "7dfd97fad21b49099ad312a68168bcef"
format_stamp: "Formatted at 2026-09-27 15:18:33 on dist-test-slave-f5h6"
server_key: "86520a397a114f67390a07f9044c6ff5"
server_key_iv: "685b898ec46a6bd90790276f3cb8713a"
server_key_version: "encryptionkey@0"
I20260927 15:18:33.360057 31981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:18:33.363749 31996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:33.365154 31981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:18:33.365326 31981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "7dfd97fad21b49099ad312a68168bcef"
format_stamp: "Formatted at 2026-09-27 15:18:33 on dist-test-slave-f5h6"
server_key: "86520a397a114f67390a07f9044c6ff5"
server_key_iv: "685b898ec46a6bd90790276f3cb8713a"
server_key_version: "encryptionkey@0"
I20260927 15:18:33.365530 31981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:33.371229 31981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:33.375528 31981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:33.375797 31981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:33.393540 31981 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40273
I20260927 15:18:33.393621 32047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40273 every 8 connection(s)
I20260927 15:18:33.395002 31981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:33.398849 32048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:33.402706 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 31981
I20260927 15:18:33.402881 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:33.403424 10373 external_mini_cluster.cc:1428] Setting key ac782013503b654d13202dd32e6645df
I20260927 15:18:33.411507 32051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:33.405351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:59763 (local address 127.0.0.1:40273)
0927 15:18:33.406133 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:33.406145 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:33.406187 (+ 42us) server_negotiation.cc:407] Connection header received
0927 15:18:33.407834 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:33.407857 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:33.408224 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:33.408571 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:33.409737 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:33.409745 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:33.409758 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:33.409777 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:33.409896 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:33.409939 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:33.410142 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:33.410252 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 15:18:33.410531 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":322,"threads_started":1}
I20260927 15:18:33.411806 32048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef: Bootstrap starting.
I20260927 15:18:33.415412 32048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:33.415684 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:18:33.416687 32048 log.cc:824] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:33.420467 32048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef: No bootstrap required, opened a new log
I20260927 15:18:33.423588 32051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:33.420948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.232:58771 (local address 127.0.0.1:40273)
0927 15:18:33.421149 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:33.421155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:33.421704 (+ 549us) server_negotiation.cc:407] Connection header received
0927 15:18:33.421833 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:33.421838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:33.421939 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:33.422057 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:33.422929 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:33.422934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:33.422938 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:33.422961 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:33.423007 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:33.423029 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:33.423152 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:33.423208 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 15:18:33.423269 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:18:33.429077 32048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfd97fad21b49099ad312a68168bcef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } }
I20260927 15:18:33.429674 32048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:33.429800 32048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dfd97fad21b49099ad312a68168bcef, State: Initialized, Role: FOLLOWER
I20260927 15:18:33.430533 32048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfd97fad21b49099ad312a68168bcef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } }
I20260927 15:18:33.430850 32048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:18:33.430997 32048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:18:33.431141 32048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:33.433521 32048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfd97fad21b49099ad312a68168bcef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } }
I20260927 15:18:33.434182 32048 leader_election.cc:302] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dfd97fad21b49099ad312a68168bcef; no voters:
I20260927 15:18:33.434816 32048 leader_election.cc:288] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:18:33.434966 32058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:33.435621 32058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef [term 1 LEADER]: Becoming Leader. State: Replica: 7dfd97fad21b49099ad312a68168bcef, State: Running, Role: LEADER
I20260927 15:18:33.436193 32058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfd97fad21b49099ad312a68168bcef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } }
I20260927 15:18:33.436817 32048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:33.439018 32060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dfd97fad21b49099ad312a68168bcef. Latest consensus state: current_term: 1 leader_uuid: "7dfd97fad21b49099ad312a68168bcef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfd97fad21b49099ad312a68168bcef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } } }
I20260927 15:18:33.439103 32059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dfd97fad21b49099ad312a68168bcef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfd97fad21b49099ad312a68168bcef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } } }
I20260927 15:18:33.439445 32060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:33.439453 32059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:33.442301 32064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:33.448360 32064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:33.457912 32064 catalog_manager.cc:1347] Generated new cluster ID: b7fbce5bee074a05b6589c286de5a93a
I20260927 15:18:33.458042 32064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:33.491781 32064 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:33.492267 32064 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:33.507642 32064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dfd97fad21b49099ad312a68168bcef: Generated new TSK 0
I20260927 15:18:33.508559 32064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:18:33.522274 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 31981
W20260927 15:18:33.523411 32043 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.232:59763
W20260927 15:18:33.524168 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 31981: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:18:33.524307 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 31981
2026-09-27T15:18:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1843 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:18:33 dist-test-slave-f5h6 krb5kdc[32079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:18:33 dist-test-slave-f5h6 krb5kdc[32079](info): set up 2 sockets
Sep 27 15:18:33 dist-test-slave-f5h6 krb5kdc[32079](info): commencing operation
krb5kdc: starting...
W20260927 15:18:34.550462 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:18:34 dist-test-slave-f5h6 krb5kdc[32079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:18:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:18:34.810711 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:34951
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:34951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:35.157375 32092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:35.157706 32092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:35.157800 32092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:35.165916 32092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:35.166079 32092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:35.166138 32092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:35.166199 32092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:35.166241 32092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:35.176919 32092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:34951
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:34951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:35.178367 32092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:35.179744 32092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:35.191751 32097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:35.191684 32098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:35.191718 32100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:35.192742 32092 server_base.cc:1034] running on GCE node
I20260927 15:18:35.193596 32092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:35.195165 32092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:35.196491 32092 hybrid_clock.cc:648] HybridClock initialized: now 1790522315196331 us; error 109 us; skew 500 ppm
I20260927 15:18:35.196851 32092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:35 dist-test-slave-f5h6 krb5kdc[32079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522315, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:35.200412 32092 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:18:35.202328 32092 webserver.cc:460] Webserver started at http://127.10.33.126:36215/ using document root <none> and password file <none>
I20260927 15:18:35.203104 32092 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:35.203239 32092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:35.203562 32092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:35.206887 32092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "0f9e112d87bb4215b0ee3661f6a29123"
format_stamp: "Formatted at 2026-09-27 15:18:35 on dist-test-slave-f5h6"
server_key: "67b4e5a76ddb5935f14b54894c1c3973"
server_key_iv: "135ed608ab2da95e9a26536a742dd65f"
server_key_version: "encryptionkey@0"
I20260927 15:18:35.207742 32092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "0f9e112d87bb4215b0ee3661f6a29123"
format_stamp: "Formatted at 2026-09-27 15:18:35 on dist-test-slave-f5h6"
server_key: "67b4e5a76ddb5935f14b54894c1c3973"
server_key_iv: "135ed608ab2da95e9a26536a742dd65f"
server_key_version: "encryptionkey@0"
I20260927 15:18:35.213582 32092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 15:18:35.217721 32107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:35.219434 32092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:18:35.219651 32092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "0f9e112d87bb4215b0ee3661f6a29123"
format_stamp: "Formatted at 2026-09-27 15:18:35 on dist-test-slave-f5h6"
server_key: "67b4e5a76ddb5935f14b54894c1c3973"
server_key_iv: "135ed608ab2da95e9a26536a742dd65f"
server_key_version: "encryptionkey@0"
I20260927 15:18:35.219872 32092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:35.234221 32092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:35.239336 32092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:35.239655 32092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:35.261257 32092 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:34951
I20260927 15:18:35.261261 32158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:34951 every 8 connection(s)
I20260927 15:18:35.262781 32092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:35.266851 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 32092
I20260927 15:18:35.267052 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:35.267181 32159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:35.267584 10373 external_mini_cluster.cc:1428] Setting key 4d9ecf8d47f1731fdb617ea366361359
I20260927 15:18:35.283022 32159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123: Bootstrap starting.
Sep 27 15:18:35 dist-test-slave-f5h6 krb5kdc[32079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:18:35.286700 32159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:35.287878 32159 log.cc:824] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:35.290856 32159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123: No bootstrap required, opened a new log
I20260927 15:18:35.292608 32162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:35.269514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34265 (local address 127.10.33.126:34951)
0927 15:18:35.270385 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:35.270397 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:35.270453 (+ 56us) server_negotiation.cc:407] Connection header received
0927 15:18:35.272252 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:35.272276 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:35.272770 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:35.273136 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:35.275033 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:35.276602 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:35.278557 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:35.279630 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:35.279800 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:35.285035 (+ 5235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:35.285045 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:35.285062 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:35.285126 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:35.288332 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:35.289255 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:35.289263 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:35.289270 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:35.289437 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:35.289940 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:35.289945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:35.289947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:35.290482 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:35.290796 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:35.291283 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 15:18:35.291613 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":409,"threads_started":1}
I20260927 15:18:35.297065 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:18:35.299517 32159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9e112d87bb4215b0ee3661f6a29123" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 34951 } }
I20260927 15:18:35.300010 32159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:35.300114 32159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9e112d87bb4215b0ee3661f6a29123, State: Initialized, Role: FOLLOWER
I20260927 15:18:35.300688 32159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9e112d87bb4215b0ee3661f6a29123" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 34951 } }
I20260927 15:18:35.300916 32159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:18:35.301066 32159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:18:35.301261 32159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:35.304073 32159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9e112d87bb4215b0ee3661f6a29123" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 34951 } }
I20260927 15:18:35.304796 32159 leader_election.cc:302] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f9e112d87bb4215b0ee3661f6a29123; no voters:
I20260927 15:18:35.305481 32159 leader_election.cc:288] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:18:35.305661 32169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:35.307072 32169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123 [term 1 LEADER]: Becoming Leader. State: Replica: 0f9e112d87bb4215b0ee3661f6a29123, State: Running, Role: LEADER
I20260927 15:18:35.307735 32169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9e112d87bb4215b0ee3661f6a29123" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 34951 } }
I20260927 15:18:35.307932 32159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:35.317883 32170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f9e112d87bb4215b0ee3661f6a29123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9e112d87bb4215b0ee3661f6a29123" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 34951 } } }
I20260927 15:18:35.317795 32171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f9e112d87bb4215b0ee3661f6a29123. Latest consensus state: current_term: 1 leader_uuid: "0f9e112d87bb4215b0ee3661f6a29123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9e112d87bb4215b0ee3661f6a29123" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 34951 } } }
I20260927 15:18:35.318326 32170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:35.318333 32171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:35.319623 32184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:35.319940 32162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:35.301356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58129 (local address 127.10.33.126:34951)
0927 15:18:35.301550 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:35.301557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:35.302045 (+ 488us) server_negotiation.cc:407] Connection header received
0927 15:18:35.302185 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:35.302191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:35.302275 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:35.302405 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:35.303959 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:35.305040 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:35.306829 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:35.307528 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:35.307676 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:35.311182 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:35.311191 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:35.311198 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:35.311255 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:35.315714 (+ 4459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:35.316617 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:35.316623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:35.316626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:35.316778 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:35.317348 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:35.317353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:35.317356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:35.318056 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:35.318356 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:35.319476 (+ 1120us) server_negotiation.cc:299] Negotiation successful
0927 15:18:35.319641 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:18:35.324925 32184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:35.334899 32184 catalog_manager.cc:1347] Generated new cluster ID: 649bc1bad80c455196da26b8da8ca7aa
I20260927 15:18:35.335062 32184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:35.355614 32184 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:35.357234 32184 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:35.379002 32184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f9e112d87bb4215b0ee3661f6a29123: Generated new TSK 0
I20260927 15:18:35.379926 32184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:18:35.390658 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 32092
I20260927 15:18:35.392002 32154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:35.488059 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 32092
W20260927 15:18:35.502863 32077 connection.cc:535] client connection to 127.10.33.126:34951 recv error: Network error: failed to read from TLS socket (remote: 127.10.33.126:34951): Connection reset by peer (error 104)
2026-09-27T15:18:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1964 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:18:35 dist-test-slave-f5h6 krb5kdc[32191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:18:35 dist-test-slave-f5h6 krb5kdc[32191](info): set up 2 sockets
Sep 27 15:18:35 dist-test-slave-f5h6 krb5kdc[32191](info): commencing operation
krb5kdc: starting...
W20260927 15:18:36.522109 10373 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:18:36 dist-test-slave-f5h6 krb5kdc[32191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:18:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:18:36.785513 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:41635
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:41635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:37.123899 32204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:37.124245 32204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:37.124343 32204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:37.132436 32204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:37.132633 32204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:37.132686 32204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:37.132742 32204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:37.132789 32204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:37.143117 32204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:41635
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:41635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:37.144433 32204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:37.145785 32204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:37.159649 32209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:37.159628 32212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:37.159591 32210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:37.166325 32204 server_base.cc:1034] running on GCE node
I20260927 15:18:37.167224 32204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:37.168840 32204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:37.170092 32204 hybrid_clock.cc:648] HybridClock initialized: now 1790522317170025 us; error 61 us; skew 500 ppm
I20260927 15:18:37.170487 32204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:37 dist-test-slave-f5h6 krb5kdc[32191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522317, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:37.181111 32204 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:18:37.182991 32204 webserver.cc:460] Webserver started at http://127.10.33.126:45127/ using document root <none> and password file <none>
I20260927 15:18:37.183784 32204 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:37.183909 32204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:37.184232 32204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:37.187546 32204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "0e009b77b1be4c70908c98db38a4e314"
format_stamp: "Formatted at 2026-09-27 15:18:37 on dist-test-slave-f5h6"
server_key: "96a2d3b1d3ce782fbe544b2150ecd94d"
server_key_iv: "7d11a594ae679525810654a5907b8165"
server_key_version: "encryptionkey@0"
I20260927 15:18:37.188647 32204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "0e009b77b1be4c70908c98db38a4e314"
format_stamp: "Formatted at 2026-09-27 15:18:37 on dist-test-slave-f5h6"
server_key: "96a2d3b1d3ce782fbe544b2150ecd94d"
server_key_iv: "7d11a594ae679525810654a5907b8165"
server_key_version: "encryptionkey@0"
I20260927 15:18:37.196044 32204 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260927 15:18:37.200009 32219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:37.201678 32204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:18:37.201915 32204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "0e009b77b1be4c70908c98db38a4e314"
format_stamp: "Formatted at 2026-09-27 15:18:37 on dist-test-slave-f5h6"
server_key: "96a2d3b1d3ce782fbe544b2150ecd94d"
server_key_iv: "7d11a594ae679525810654a5907b8165"
server_key_version: "encryptionkey@0"
I20260927 15:18:37.202126 32204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:37.225559 32204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:37.230612 32204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:37.230922 32204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:37.250198 32204 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:41635
I20260927 15:18:37.250216 32270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:41635 every 8 connection(s)
I20260927 15:18:37.251785 32204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:37.252233 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 32204
I20260927 15:18:37.252429 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:37.252975 10373 external_mini_cluster.cc:1428] Setting key bc88f99bf9e45205947e610b7ac6f367
I20260927 15:18:37.256665 32271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:37.270020 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:37.255103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34065 (local address 127.10.33.126:41635)
0927 15:18:37.256015 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:37.256034 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:37.256086 (+ 52us) server_negotiation.cc:407] Connection header received
0927 15:18:37.258625 (+ 2539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:37.258660 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:37.259224 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:37.259726 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:37.260639 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:37.262775 (+ 2136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:37.264393 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:37.265473 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:37.265671 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:37.266992 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:37.267003 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:37.267024 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:37.267794 (+ 770us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:37.267899 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:37.267940 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:37.268121 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:37.268380 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 15:18:37.268597 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":398,"threads_started":1}
I20260927 15:18:37.271314 32271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314: Bootstrap starting.
I20260927 15:18:37.274536 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:18:37.275066 32271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:37.276096 32271 log.cc:824] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:37.279742 32271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314: No bootstrap required, opened a new log
I20260927 15:18:37.286260 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:37.279236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52661 (local address 127.10.33.126:41635)
0927 15:18:37.279420 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:37.279426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:37.279805 (+ 379us) server_negotiation.cc:407] Connection header received
0927 15:18:37.280030 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:37.280035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:37.280116 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:37.280229 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:37.280720 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:37.281836 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:37.283284 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:37.283981 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:37.284109 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:37.285295 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:37.285302 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:37.285308 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:37.285332 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:37.285387 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:37.285409 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:37.285557 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:37.285813 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 15:18:37.285914 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:18:37.289269 32271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e009b77b1be4c70908c98db38a4e314" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 41635 } }
I20260927 15:18:37.289688 32271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:37.289769 32271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e009b77b1be4c70908c98db38a4e314, State: Initialized, Role: FOLLOWER
I20260927 15:18:37.290668 32271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e009b77b1be4c70908c98db38a4e314" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 41635 } }
I20260927 15:18:37.290935 32271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:18:37.291159 32271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:18:37.291363 32271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:37.293617 32271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e009b77b1be4c70908c98db38a4e314" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 41635 } }
I20260927 15:18:37.294193 32271 leader_election.cc:302] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e009b77b1be4c70908c98db38a4e314; no voters:
I20260927 15:18:37.294775 32271 leader_election.cc:288] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:18:37.294912 32281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:37.295627 32281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314 [term 1 LEADER]: Becoming Leader. State: Replica: 0e009b77b1be4c70908c98db38a4e314, State: Running, Role: LEADER
I20260927 15:18:37.296186 32281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e009b77b1be4c70908c98db38a4e314" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 41635 } }
I20260927 15:18:37.296815 32271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:37.299144 32282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e009b77b1be4c70908c98db38a4e314" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e009b77b1be4c70908c98db38a4e314" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 41635 } } }
I20260927 15:18:37.299906 32282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:37.299469 32283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e009b77b1be4c70908c98db38a4e314. Latest consensus state: current_term: 1 leader_uuid: "0e009b77b1be4c70908c98db38a4e314" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e009b77b1be4c70908c98db38a4e314" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 41635 } } }
I20260927 15:18:37.300267 32283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:37.301216 32286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:37.306915 32286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:37.316636 32286 catalog_manager.cc:1347] Generated new cluster ID: 63babb1dd9964695ae4b5e0012c5ecae
I20260927 15:18:37.316759 32286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:37.330437 32286 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:37.332369 32286 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:37.348464 32286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e009b77b1be4c70908c98db38a4e314: Generated new TSK 0
I20260927 15:18:37.349488 32286 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:18:37.355810 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 32204
W20260927 15:18:37.357107 32266 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34065
W20260927 15:18:37.357964 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 32204: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:18:37.358110 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 32204
W20260927 15:18:37.372558 32189 connection.cc:535] client connection to 127.10.33.126:41635 recv error: Network error: failed to read from TLS socket (remote: 127.10.33.126:41635): Connection reset by peer (error 104)
2026-09-27T15:18:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1868 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:18:37 dist-test-slave-f5h6 krb5kdc[32302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:18:37 dist-test-slave-f5h6 krb5kdc[32302](info): set up 2 sockets
Sep 27 15:18:37 dist-test-slave-f5h6 krb5kdc[32302](info): commencing operation
krb5kdc: starting...
W20260927 15:18:38.410284 10373 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:18:38 dist-test-slave-f5h6 krb5kdc[32302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:18:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:18:38.661041 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:35495
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:35495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:38.994230 32315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:38.994589 32315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:38.994699 32315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:39.001983 32315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:39.002107 32315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:39.002152 32315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:39.002211 32315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:39.002254 32315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:39.012499 32315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:35495
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:35495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:39.013974 32315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:39.015290 32315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:39.027000 32320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:39.027311 32321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:39.027493 32323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:39.027807 32315 server_base.cc:1034] running on GCE node
I20260927 15:18:39.028831 32315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:39.030406 32315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:39.031630 32315 hybrid_clock.cc:648] HybridClock initialized: now 1790522319031583 us; error 31 us; skew 500 ppm
I20260927 15:18:39.031984 32315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:39 dist-test-slave-f5h6 krb5kdc[32302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522319, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:39.035099 32315 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:18:39.036585 32315 webserver.cc:460] Webserver started at http://127.10.33.126:40251/ using document root <none> and password file <none>
I20260927 15:18:39.037263 32315 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:39.037369 32315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:39.037664 32315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:39.040755 32315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "b2fb5bdb134244cfb5966140da8f30e4"
format_stamp: "Formatted at 2026-09-27 15:18:39 on dist-test-slave-f5h6"
server_key: "1001024f1f1e08bfc7dd4aaa5b33cd8b"
server_key_iv: "1317e8c5f11cbc98189de729976143a8"
server_key_version: "encryptionkey@0"
I20260927 15:18:39.041507 32315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "b2fb5bdb134244cfb5966140da8f30e4"
format_stamp: "Formatted at 2026-09-27 15:18:39 on dist-test-slave-f5h6"
server_key: "1001024f1f1e08bfc7dd4aaa5b33cd8b"
server_key_iv: "1317e8c5f11cbc98189de729976143a8"
server_key_version: "encryptionkey@0"
I20260927 15:18:39.046757 32315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260927 15:18:39.050164 32330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:39.051492 32315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:18:39.051647 32315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "b2fb5bdb134244cfb5966140da8f30e4"
format_stamp: "Formatted at 2026-09-27 15:18:39 on dist-test-slave-f5h6"
server_key: "1001024f1f1e08bfc7dd4aaa5b33cd8b"
server_key_iv: "1317e8c5f11cbc98189de729976143a8"
server_key_version: "encryptionkey@0"
I20260927 15:18:39.051831 32315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:39.057248 32315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:39.061447 32315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:39.061774 32315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:39.079399 32315 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:35495
I20260927 15:18:39.079463 32381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:35495 every 8 connection(s)
I20260927 15:18:39.080758 32315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:39.084494 32382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:39.086367 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 32315
I20260927 15:18:39.086553 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:39.087113 10373 external_mini_cluster.cc:1428] Setting key 3a2b286535342295edf760807119e7a1
I20260927 15:18:39.095376 32385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:39.089131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46517 (local address 127.10.33.126:35495)
0927 15:18:39.090044 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:39.090057 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:39.090112 (+ 55us) server_negotiation.cc:407] Connection header received
0927 15:18:39.091770 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:39.091797 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:39.092227 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:39.092591 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:39.093766 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:39.093773 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:39.093786 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:39.093802 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:39.093901 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:39.093927 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:39.094133 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:39.094230 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 15:18:39.094432 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":387,"threads_started":1}
I20260927 15:18:39.099615 32382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4: Bootstrap starting.
I20260927 15:18:39.099797 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:18:39.102807 32382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:39.103909 32382 log.cc:824] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:39.106550 32385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:39.104097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57175 (local address 127.10.33.126:35495)
0927 15:18:39.104290 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:39.104296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:39.104678 (+ 382us) server_negotiation.cc:407] Connection header received
0927 15:18:39.104849 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:39.104853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:39.104906 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:39.104996 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:39.105898 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:39.105916 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:39.105921 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:39.105938 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:39.105981 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:39.105997 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:39.106103 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:39.106229 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 15:18:39.106291 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:18:39.107004 32382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4: No bootstrap required, opened a new log
I20260927 15:18:39.115010 32382 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2fb5bdb134244cfb5966140da8f30e4" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35495 } }
I20260927 15:18:39.115404 32382 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:39.115476 32382 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2fb5bdb134244cfb5966140da8f30e4, State: Initialized, Role: FOLLOWER
I20260927 15:18:39.116039 32382 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2fb5bdb134244cfb5966140da8f30e4" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35495 } }
I20260927 15:18:39.116253 32382 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:18:39.116353 32382 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:18:39.116475 32382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:39.119084 32382 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2fb5bdb134244cfb5966140da8f30e4" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35495 } }
I20260927 15:18:39.119581 32382 leader_election.cc:302] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2fb5bdb134244cfb5966140da8f30e4; no voters:
I20260927 15:18:39.120182 32382 leader_election.cc:288] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:18:39.120311 32392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:39.120936 32392 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4 [term 1 LEADER]: Becoming Leader. State: Replica: b2fb5bdb134244cfb5966140da8f30e4, State: Running, Role: LEADER
I20260927 15:18:39.121511 32392 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2fb5bdb134244cfb5966140da8f30e4" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35495 } }
I20260927 15:18:39.122292 32382 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:39.124513 32394 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2fb5bdb134244cfb5966140da8f30e4. Latest consensus state: current_term: 1 leader_uuid: "b2fb5bdb134244cfb5966140da8f30e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2fb5bdb134244cfb5966140da8f30e4" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35495 } } }
I20260927 15:18:39.124812 32393 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2fb5bdb134244cfb5966140da8f30e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2fb5bdb134244cfb5966140da8f30e4" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 35495 } } }
I20260927 15:18:39.125033 32394 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:39.125077 32393 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:39.125967 32401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:39.132251 32401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:39.141573 32401 catalog_manager.cc:1347] Generated new cluster ID: 36cd1c0bb2d6482d8a49aa70862e9175
I20260927 15:18:39.141702 32401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:39.169835 32401 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:39.170315 32401 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:39.177695 32401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2fb5bdb134244cfb5966140da8f30e4: Generated new TSK 0
I20260927 15:18:39.178584 32401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:18:39.184190 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 32315
W20260927 15:18:39.185308 32377 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46517
W20260927 15:18:39.186061 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 32315: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:18:39.186194 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 32315
2026-09-27T15:18:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1827 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:18:39 dist-test-slave-f5h6 krb5kdc[32413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:18:39 dist-test-slave-f5h6 krb5kdc[32413](info): set up 2 sockets
Sep 27 15:18:39 dist-test-slave-f5h6 krb5kdc[32413](info): commencing operation
krb5kdc: starting...
W20260927 15:18:40.214251 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:18:40 dist-test-slave-f5h6 krb5kdc[32413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:18:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:18:40.481676 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:45235
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:45235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:40.816001 32426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:40.816340 32426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:40.816464 32426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:40.823829 32426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:40.823968 32426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:40.824038 32426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:40.824117 32426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:40.824175 32426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:40.834177 32426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:45235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:45235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:40.835541 32426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:40.836848 32426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:40.848647 32431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:40.848666 32434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:40.848699 32432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:40.849468 32426 server_base.cc:1034] running on GCE node
I20260927 15:18:40.850268 32426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:40.851752 32426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:40.852984 32426 hybrid_clock.cc:648] HybridClock initialized: now 1790522320852930 us; error 41 us; skew 500 ppm
I20260927 15:18:40.853340 32426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:40 dist-test-slave-f5h6 krb5kdc[32413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522320, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:40.856411 32426 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:18:40.858054 32426 webserver.cc:460] Webserver started at http://127.10.33.126:35605/ using document root <none> and password file <none>
I20260927 15:18:40.858728 32426 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:40.858834 32426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:40.859112 32426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:40.862087 32426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "60b63d8f652d45ebb59f0ff1b6c033af"
format_stamp: "Formatted at 2026-09-27 15:18:40 on dist-test-slave-f5h6"
server_key: "d7ad6b2e9c4364c90ae315fbec7d362b"
server_key_iv: "6fa4a0784b11e6c0f16ea7b4bbc9ef36"
server_key_version: "encryptionkey@0"
I20260927 15:18:40.862843 32426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "60b63d8f652d45ebb59f0ff1b6c033af"
format_stamp: "Formatted at 2026-09-27 15:18:40 on dist-test-slave-f5h6"
server_key: "d7ad6b2e9c4364c90ae315fbec7d362b"
server_key_iv: "6fa4a0784b11e6c0f16ea7b4bbc9ef36"
server_key_version: "encryptionkey@0"
I20260927 15:18:40.868237 32426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 15:18:40.872226 32441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:40.873643 32426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:18:40.873807 32426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "60b63d8f652d45ebb59f0ff1b6c033af"
format_stamp: "Formatted at 2026-09-27 15:18:40 on dist-test-slave-f5h6"
server_key: "d7ad6b2e9c4364c90ae315fbec7d362b"
server_key_iv: "6fa4a0784b11e6c0f16ea7b4bbc9ef36"
server_key_version: "encryptionkey@0"
I20260927 15:18:40.874043 32426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:40.892467 32426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:40.897297 32426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:40.897596 32426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:40.916066 32426 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:45235
I20260927 15:18:40.916087 32492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:45235 every 8 connection(s)
I20260927 15:18:40.917554 32426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:40.921638 32493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:40.926795 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 32426
I20260927 15:18:40.926960 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:40.927407 10373 external_mini_cluster.cc:1428] Setting key fd874104b6694ee320c93fd1c6571c01
I20260927 15:18:40.932924 32493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af: Bootstrap starting.
I20260927 15:18:40.936868 32493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:40.938203 32493 log.cc:824] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:40.941432 32493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af: No bootstrap required, opened a new log
Sep 27 15:18:40 dist-test-slave-f5h6 krb5kdc[32413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:18:40.952744 32493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b63d8f652d45ebb59f0ff1b6c033af" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45235 } }
I20260927 15:18:40.953217 32496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:40.929185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42193 (local address 127.10.33.126:45235)
0927 15:18:40.930077 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:40.930091 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:40.930137 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:18:40.931907 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:40.931932 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:40.932370 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:40.932727 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:40.934824 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:40.936237 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:40.938022 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:40.939029 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:40.939273 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:40.943118 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:40.943132 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:40.943155 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:40.943219 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:40.947788 (+ 4569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:40.948674 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:40.948686 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:40.948702 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:40.948975 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:40.949387 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:40.949394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:40.949398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:40.950288 (+ 890us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:40.950627 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:40.951211 (+ 584us) server_negotiation.cc:299] Negotiation successful
0927 15:18:40.951751 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":385,"threads_started":1}
I20260927 15:18:40.953253 32493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:40.953348 32493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b63d8f652d45ebb59f0ff1b6c033af, State: Initialized, Role: FOLLOWER
I20260927 15:18:40.954069 32493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b63d8f652d45ebb59f0ff1b6c033af" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45235 } }
I20260927 15:18:40.954339 32493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:18:40.954486 32493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:18:40.954679 32493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:40.957367 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:18:40.957401 32493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b63d8f652d45ebb59f0ff1b6c033af" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45235 } }
I20260927 15:18:40.958061 32493 leader_election.cc:302] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60b63d8f652d45ebb59f0ff1b6c033af; no voters:
I20260927 15:18:40.958773 32493 leader_election.cc:288] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:18:40.958882 32498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:40.959972 32498 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af [term 1 LEADER]: Becoming Leader. State: Replica: 60b63d8f652d45ebb59f0ff1b6c033af, State: Running, Role: LEADER
I20260927 15:18:40.960525 32498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b63d8f652d45ebb59f0ff1b6c033af" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45235 } }
I20260927 15:18:40.961272 32493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:40.963361 32500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60b63d8f652d45ebb59f0ff1b6c033af. Latest consensus state: current_term: 1 leader_uuid: "60b63d8f652d45ebb59f0ff1b6c033af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b63d8f652d45ebb59f0ff1b6c033af" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45235 } } }
I20260927 15:18:40.963668 32500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:40.964097 32499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60b63d8f652d45ebb59f0ff1b6c033af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b63d8f652d45ebb59f0ff1b6c033af" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 45235 } } }
I20260927 15:18:40.964397 32499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:40.964505 32508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:40.983261 32508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:40.989285 32496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:40.962967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51083 (local address 127.10.33.126:45235)
0927 15:18:40.963173 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:40.963180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:40.966846 (+ 3666us) server_negotiation.cc:407] Connection header received
0927 15:18:40.967494 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:40.967502 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:40.967606 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:40.967693 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:40.970717 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:40.972064 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:40.974398 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:40.975261 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:40.975388 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:40.979565 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:40.979575 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:40.979581 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:40.979642 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:40.984254 (+ 4612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:40.985393 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:40.985401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:40.985405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:40.985606 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:40.986191 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:40.986198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:40.986202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:40.986744 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:40.987074 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:40.988491 (+ 1417us) server_negotiation.cc:299] Negotiation successful
0927 15:18:40.988768 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 15:18:40.996274 32508 catalog_manager.cc:1347] Generated new cluster ID: fe9bd6110aa246f8bde04760a2676862
I20260927 15:18:40.996411 32508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:41.034910 32508 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:41.036736 32508 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:41.054467 32508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60b63d8f652d45ebb59f0ff1b6c033af: Generated new TSK 0
I20260927 15:18:41.055444 32508 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:18:41.062027 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 32426
I20260927 15:18:41.063164 32488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:41.159564 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 32426
2026-09-27T15:18:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1975 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:18:41 dist-test-slave-f5h6 krb5kdc[32525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:18:41 dist-test-slave-f5h6 krb5kdc[32525](info): set up 2 sockets
Sep 27 15:18:41 dist-test-slave-f5h6 krb5kdc[32525](info): commencing operation
krb5kdc: starting...
W20260927 15:18:42.222563 10373 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:18:42 dist-test-slave-f5h6 krb5kdc[32525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:18:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:18:42.486971 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:42651
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:34029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:42651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:42.843812 32538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:42.844129 32538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:42.844250 32538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:42.851858 32538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:42.851979 32538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:42.852026 32538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:42.852108 32538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:42.852173 32538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:42.863658 32538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:34029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:42651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:42651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:42.865077 32538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:42.866640 32538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:42.879565 32543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:42.879537 32544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:42.879628 32546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:42.880537 32538 server_base.cc:1034] running on GCE node
I20260927 15:18:42.881400 32538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:42.883016 32538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:42.884259 32538 hybrid_clock.cc:648] HybridClock initialized: now 1790522322884208 us; error 48 us; skew 500 ppm
I20260927 15:18:42.884629 32538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:42 dist-test-slave-f5h6 krb5kdc[32525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522322, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:42.893306 32538 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:18:42.895062 32538 webserver.cc:460] Webserver started at http://127.10.33.126:42711/ using document root <none> and password file <none>
I20260927 15:18:42.896009 32538 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:42.896142 32538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:42.896437 32538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:42.899824 32538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "b56b4f0d2a5744e9b3d09d0e8fed68e0"
format_stamp: "Formatted at 2026-09-27 15:18:42 on dist-test-slave-f5h6"
server_key: "e44e509c088bbf5fb4d09b194ec933ee"
server_key_iv: "b99d4562d3d3be43fba3eec1e0585de4"
server_key_version: "encryptionkey@0"
I20260927 15:18:42.900697 32538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "b56b4f0d2a5744e9b3d09d0e8fed68e0"
format_stamp: "Formatted at 2026-09-27 15:18:42 on dist-test-slave-f5h6"
server_key: "e44e509c088bbf5fb4d09b194ec933ee"
server_key_iv: "b99d4562d3d3be43fba3eec1e0585de4"
server_key_version: "encryptionkey@0"
I20260927 15:18:42.906934 32538 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 15:18:42.911149 32553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:42.912743 32538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:18:42.912919 32538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "b56b4f0d2a5744e9b3d09d0e8fed68e0"
format_stamp: "Formatted at 2026-09-27 15:18:42 on dist-test-slave-f5h6"
server_key: "e44e509c088bbf5fb4d09b194ec933ee"
server_key_iv: "b99d4562d3d3be43fba3eec1e0585de4"
server_key_version: "encryptionkey@0"
I20260927 15:18:42.913106 32538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:42.938930 32538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:42.944712 32538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:42.945017 32538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:42.964738 32538 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:42651
I20260927 15:18:42.964794 32604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:42651 every 8 connection(s)
I20260927 15:18:42.966352 32538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:42.970393 32605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:42.974815 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 32538
I20260927 15:18:42.975020 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:42.975520 10373 external_mini_cluster.cc:1428] Setting key ce647ab622a195759efab13364e319c4
I20260927 15:18:42.984227 32605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0: Bootstrap starting.
I20260927 15:18:42.988165 32605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:42.989466 32605 log.cc:824] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:42.990556 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:42.977527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59191 (local address 127.10.33.126:42651)
0927 15:18:42.978344 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:42.978357 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:42.978400 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:18:42.980061 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:42.980088 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:42.980548 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:42.980917 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:42.981510 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:42.982981 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:42.984823 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:42.985559 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:42.985728 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:42.987429 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:42.987441 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:42.987459 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:42.988166 (+ 707us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:42.988270 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:42.988307 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:42.988430 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:42.988754 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 15:18:42.989076 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":377,"threads_started":1}
I20260927 15:18:42.993011 32605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0: No bootstrap required, opened a new log
I20260927 15:18:42.994974 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:18:43.003651 32605 raft_consensus.cc:348] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56b4f0d2a5744e9b3d09d0e8fed68e0" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42651 } }
I20260927 15:18:43.004216 32605 raft_consensus.cc:374] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:43.004330 32605 raft_consensus.cc:729] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b56b4f0d2a5744e9b3d09d0e8fed68e0, State: Initialized, Role: FOLLOWER
I20260927 15:18:43.005069 32605 consensus_queue.cc:260] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56b4f0d2a5744e9b3d09d0e8fed68e0" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42651 } }
I20260927 15:18:43.005378 32605 raft_consensus.cc:388] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:18:43.005508 32605 raft_consensus.cc:482] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:18:43.005687 32605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:43.008188 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:43.000678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42385 (local address 127.10.33.126:42651)
0927 15:18:43.000893 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:43.000899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:43.001273 (+ 374us) server_negotiation.cc:407] Connection header received
0927 15:18:43.001429 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:43.001433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:43.001506 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:43.001612 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:43.002316 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:43.003061 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:43.005032 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:43.005810 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:43.005977 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:43.007369 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:43.007375 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:43.007381 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:43.007404 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:43.007463 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:43.007483 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:43.007571 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:43.007720 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 15:18:43.007817 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 15:18:43.008939 32605 raft_consensus.cc:504] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56b4f0d2a5744e9b3d09d0e8fed68e0" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42651 } }
I20260927 15:18:43.009511 32605 leader_election.cc:302] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b56b4f0d2a5744e9b3d09d0e8fed68e0; no voters:
I20260927 15:18:43.010196 32605 leader_election.cc:288] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:18:43.010347 32615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:43.010968 32615 raft_consensus.cc:686] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0 [term 1 LEADER]: Becoming Leader. State: Replica: b56b4f0d2a5744e9b3d09d0e8fed68e0, State: Running, Role: LEADER
I20260927 15:18:43.011536 32615 consensus_queue.cc:237] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56b4f0d2a5744e9b3d09d0e8fed68e0" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42651 } }
I20260927 15:18:43.012279 32605 sys_catalog.cc:564] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:43.015504 32616 sys_catalog.cc:455] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b56b4f0d2a5744e9b3d09d0e8fed68e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56b4f0d2a5744e9b3d09d0e8fed68e0" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42651 } } }
I20260927 15:18:43.016042 32616 sys_catalog.cc:458] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:43.015496 32617 sys_catalog.cc:455] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b56b4f0d2a5744e9b3d09d0e8fed68e0. Latest consensus state: current_term: 1 leader_uuid: "b56b4f0d2a5744e9b3d09d0e8fed68e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56b4f0d2a5744e9b3d09d0e8fed68e0" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 42651 } } }
I20260927 15:18:43.017007 32617 sys_catalog.cc:458] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:43.017818 32624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:43.024231 32624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:43.035907 32624 catalog_manager.cc:1347] Generated new cluster ID: 5bc6864f92374220a85049dba89063bc
I20260927 15:18:43.036044 32624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:43.054201 32624 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:43.055516 32624 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:43.064199 32624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b56b4f0d2a5744e9b3d09d0e8fed68e0: Generated new TSK 0
I20260927 15:18:43.065080 32624 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:18:43.078068 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 32538
W20260927 15:18:43.079553 32600 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59191
W20260927 15:18:43.080525 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 32538: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:18:43.080683 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 32538
2026-09-27T15:18:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1919 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:18:43 dist-test-slave-f5h6 krb5kdc[32636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:18:43 dist-test-slave-f5h6 krb5kdc[32636](info): set up 2 sockets
Sep 27 15:18:43 dist-test-slave-f5h6 krb5kdc[32636](info): commencing operation
krb5kdc: starting...
W20260927 15:18:44.122345 10373 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:18:44 dist-test-slave-f5h6 krb5kdc[32636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:18:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:18:44.389147 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:44047
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:45513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:44047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:44.722469 32649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:44.722787 32649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:44.722911 32649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:44.730393 32649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:44.730520 32649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:44.730571 32649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:44.730652 32649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:44.730707 32649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:44.740725 32649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:45513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:44047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:44047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:44.742081 32649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:44.743350 32649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:44.754925 32654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:44.754935 32655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:44.754976 32657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:44.755880 32649 server_base.cc:1034] running on GCE node
I20260927 15:18:44.756691 32649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:44.758339 32649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:44.759577 32649 hybrid_clock.cc:648] HybridClock initialized: now 1790522324759537 us; error 88 us; skew 500 ppm
I20260927 15:18:44.759981 32649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:44 dist-test-slave-f5h6 krb5kdc[32636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522324, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:44.763211 32649 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:18:44.764814 32649 webserver.cc:460] Webserver started at http://127.10.33.126:45663/ using document root <none> and password file <none>
I20260927 15:18:44.765514 32649 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:44.765643 32649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:44.765993 32649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:44.769130 32649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "79b4666df9254555ab2598351b2b52f6"
format_stamp: "Formatted at 2026-09-27 15:18:44 on dist-test-slave-f5h6"
server_key: "215477c28d24addcf33accd0b05df7c8"
server_key_iv: "52530322aa3eec8aeae748136e1e5401"
server_key_version: "encryptionkey@0"
I20260927 15:18:44.770056 32649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "79b4666df9254555ab2598351b2b52f6"
format_stamp: "Formatted at 2026-09-27 15:18:44 on dist-test-slave-f5h6"
server_key: "215477c28d24addcf33accd0b05df7c8"
server_key_iv: "52530322aa3eec8aeae748136e1e5401"
server_key_version: "encryptionkey@0"
I20260927 15:18:44.775669 32649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:18:44.779623 32664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:44.781113 32649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:18:44.781294 32649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "79b4666df9254555ab2598351b2b52f6"
format_stamp: "Formatted at 2026-09-27 15:18:44 on dist-test-slave-f5h6"
server_key: "215477c28d24addcf33accd0b05df7c8"
server_key_iv: "52530322aa3eec8aeae748136e1e5401"
server_key_version: "encryptionkey@0"
I20260927 15:18:44.781495 32649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:44.787668 32649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:44.791960 32649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:44.792425 32649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:44.810402 32649 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:44047
I20260927 15:18:44.810452 32715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:44047 every 8 connection(s)
I20260927 15:18:44.811839 32649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:44.814183 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 32649
I20260927 15:18:44.814347 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:44.814757 10373 external_mini_cluster.cc:1428] Setting key 0b7e5de8a70e87f6d910e6fa9a77dde2
I20260927 15:18:44.816493 32716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:44.824115 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:44.816846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51777 (local address 127.10.33.126:44047)
0927 15:18:44.817685 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:44.817701 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:44.817765 (+ 64us) server_negotiation.cc:407] Connection header received
0927 15:18:44.819829 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:44.819861 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:44.820402 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:44.820915 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:44.822006 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:44.822015 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:44.822034 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:44.822051 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:44.822140 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:44.822180 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:44.822446 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:44.822584 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 15:18:44.822816 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":353,"threads_started":1}
I20260927 15:18:44.828119 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:18:44.830425 32716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6: Bootstrap starting.
I20260927 15:18:44.833773 32716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:44.834625 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:44.832427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38815 (local address 127.10.33.126:44047)
0927 15:18:44.832606 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:44.832612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:44.832879 (+ 267us) server_negotiation.cc:407] Connection header received
0927 15:18:44.833018 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:44.833022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:44.833084 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:44.833194 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:44.834011 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:44.834017 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:44.834022 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:18:44.834049 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:44.834093 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:18:44.834112 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:44.834223 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:44.834303 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 15:18:44.834363 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:18:44.835001 32716 log.cc:824] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:44.837742 32716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6: No bootstrap required, opened a new log
I20260927 15:18:44.845034 32716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b4666df9254555ab2598351b2b52f6" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44047 } }
I20260927 15:18:44.845492 32716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:44.845589 32716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b4666df9254555ab2598351b2b52f6, State: Initialized, Role: FOLLOWER
I20260927 15:18:44.846254 32716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b4666df9254555ab2598351b2b52f6" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44047 } }
I20260927 15:18:44.846501 32716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:18:44.846642 32716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:18:44.846800 32716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:44.849352 32716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b4666df9254555ab2598351b2b52f6" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44047 } }
I20260927 15:18:44.850014 32716 leader_election.cc:302] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79b4666df9254555ab2598351b2b52f6; no voters:
I20260927 15:18:44.850705 32716 leader_election.cc:288] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:18:44.850829 32726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:44.851498 32726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6 [term 1 LEADER]: Becoming Leader. State: Replica: 79b4666df9254555ab2598351b2b52f6, State: Running, Role: LEADER
I20260927 15:18:44.852104 32726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b4666df9254555ab2598351b2b52f6" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44047 } }
I20260927 15:18:44.852735 32716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:44.855077 32727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79b4666df9254555ab2598351b2b52f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b4666df9254555ab2598351b2b52f6" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44047 } } }
I20260927 15:18:44.855746 32727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:44.855132 32728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79b4666df9254555ab2598351b2b52f6. Latest consensus state: current_term: 1 leader_uuid: "79b4666df9254555ab2598351b2b52f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b4666df9254555ab2598351b2b52f6" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44047 } } }
I20260927 15:18:44.856029 32728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:44.857004 32735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:44.862954 32735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:44.873207 32735 catalog_manager.cc:1347] Generated new cluster ID: f4d94172dd7f44c79dfed45a95c77bf3
I20260927 15:18:44.873343 32735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:44.896052 32735 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:44.896605 32735 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:44.903468 32735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79b4666df9254555ab2598351b2b52f6: Generated new TSK 0
I20260927 15:18:44.904299 32735 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:18:44.911682 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 32649
W20260927 15:18:44.912914 32711 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51777
W20260927 15:18:44.913576 10373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 32649: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 15:18:44.913697 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 32649
2026-09-27T15:18:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1832 ms)
[----------] 18 tests from AuthTokenIssuingTest (33840 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:18:45 dist-test-slave-f5h6 krb5kdc[32747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:18:45 dist-test-slave-f5h6 krb5kdc[32747](info): set up 2 sockets
Sep 27 15:18:45 dist-test-slave-f5h6 krb5kdc[32747](info): commencing operation
krb5kdc: starting...
W20260927 15:18:45.942353 10373 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:18:46 dist-test-slave-f5h6 krb5kdc[32747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:18:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:18:46.205448 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:46507
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:33409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:46507,127.10.33.125:39831,127.10.33.124:41433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:46.538384 32760 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:18:46.538761 32760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:46.538864 32760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:46.538964 32760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:46.546367 32760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:46.546478 32760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:46.546535 32760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:46.546584 32760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:46.546645 32760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:46.556787 32760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:33409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:46507,127.10.33.125:39831,127.10.33.124:41433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:46507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:46.558117 32760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:46.559504 32760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:46.577229 32765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:46.577198 300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:46.577279 32766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:46.578682 32760 server_base.cc:1034] running on GCE node
I20260927 15:18:46.579618 32760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:46.581171 32760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:46.582396 32760 hybrid_clock.cc:648] HybridClock initialized: now 1790522326582331 us; error 46 us; skew 500 ppm
I20260927 15:18:46.582728 32760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:46 dist-test-slave-f5h6 krb5kdc[32747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522326, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:46.585933 32760 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:18:46.587438 32760 webserver.cc:460] Webserver started at http://127.10.33.126:34009/ using document root <none> and password file <none>
I20260927 15:18:46.588145 32760 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:46.588251 32760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:46.588581 32760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:46.591838 32760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "64541b67c04e43dd93642309f5643a4d"
format_stamp: "Formatted at 2026-09-27 15:18:46 on dist-test-slave-f5h6"
server_key: "1295d900ef2339e1c74865ef54302b4c"
server_key_iv: "4ebfd1b6220a78edc9155fd06ef3a9f0"
server_key_version: "encryptionkey@0"
I20260927 15:18:46.592684 32760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "64541b67c04e43dd93642309f5643a4d"
format_stamp: "Formatted at 2026-09-27 15:18:46 on dist-test-slave-f5h6"
server_key: "1295d900ef2339e1c74865ef54302b4c"
server_key_iv: "4ebfd1b6220a78edc9155fd06ef3a9f0"
server_key_version: "encryptionkey@0"
I20260927 15:18:46.598140 32760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 15:18:46.601763 307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:46.603169 32760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:18:46.603356 32760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "64541b67c04e43dd93642309f5643a4d"
format_stamp: "Formatted at 2026-09-27 15:18:46 on dist-test-slave-f5h6"
server_key: "1295d900ef2339e1c74865ef54302b4c"
server_key_iv: "4ebfd1b6220a78edc9155fd06ef3a9f0"
server_key_version: "encryptionkey@0"
I20260927 15:18:46.603571 32760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:46.624773 32760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:46.629300 32760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:46.629577 32760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:46.647864 32760 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:46507
I20260927 15:18:46.647920 358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:46507 every 8 connection(s)
I20260927 15:18:46.649235 32760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:46.651011 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 32760
I20260927 15:18:46.651211 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:46.651863 10373 external_mini_cluster.cc:1428] Setting key 38bff32ac50913cbed624fc57e1a0166
I20260927 15:18:46.653206 359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:46.659946 359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46507 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.33.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.125@KRBTEST.COM" created.
Sep 27 15:18:46 dist-test-slave-f5h6 krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522326, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:18:46.686929 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:46.664101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:46507 (local address 127.0.0.1:54482)
0927 15:18:46.665423 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:46.665467 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:46.667372 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:46.668388 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:46.668393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:46.668995 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:46.669370 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:46.669389 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:46.671569 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:46.671573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:46.672706 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:46.672720 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:46.674090 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:46.674096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:46.674363 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:46.675053 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:46.675083 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:46.679019 (+ 3936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:46.682482 (+ 3463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:46.682495 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:46.682508 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:46.683036 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:46.683476 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:46.683481 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:46.683484 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:46.683672 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:46.684533 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:46.684541 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:46.684774 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 15:18:46.685036 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 15:18:46.685753 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"mutex_wait_us":267,"spinlock_wait_cycles":69504,"thread_start_us":813,"threads_started":1}
I20260927 15:18:46.686904 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:46.664101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54482 (local address 127.10.33.126:46507)
0927 15:18:46.665007 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:46.665023 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:46.665619 (+ 596us) server_negotiation.cc:407] Connection header received
0927 15:18:46.667811 (+ 2192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:46.667818 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:46.668129 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:46.668345 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:46.669604 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:46.671400 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:46.672880 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:46.673798 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:46.674019 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:46.679359 (+ 5340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:46.679372 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:46.679385 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:46.679424 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:46.682331 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:46.683176 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:46.683184 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:46.683187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:46.683357 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:46.683768 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:46.683771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:46.683774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:46.684385 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:46.684540 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:46.685160 (+ 620us) server_negotiation.cc:299] Negotiation successful
0927 15:18:46.685583 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":665,"thread_start_us":417,"threads_started":1}
I20260927 15:18:46.690336 359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:18:46.691520 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:46.691059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39831 (local address 127.0.0.1:47368)
0927 15:18:46.691234 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:46.691367 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 15:18:46.691821 309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39831: connect: Connection refused (error 111)
I20260927 15:18:46.693485 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:46.693056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39831 (local address 127.0.0.1:47380)
0927 15:18:46.693224 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:46.693310 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 15:18:46.693722 359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.125:39831: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39831: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.125.keytab.
Entry for principal kudu/127.10.33.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.125.keytab.
I20260927 15:18:46.711105 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.125:39831
--webserver_interface=127.10.33.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.125.keytab
--principal=kudu/127.10.33.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:33409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:46507,127.10.33.125:39831,127.10.33.124:41433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:18:46.743122 359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } attempt: 1
I20260927 15:18:46.744259 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:46.743802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39831 (local address 127.0.0.1:47394)
0927 15:18:46.744007 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:46.744132 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 15:18:46.745174 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:46.744840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39831 (local address 127.0.0.1:47404)
0927 15:18:46.744979 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:46.745044 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 15:18:46.745344 359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.125:39831: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39831: connect: Connection refused (error 111)
I20260927 15:18:46.813593 359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } attempt: 2
I20260927 15:18:46.814821 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:46.814290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39831 (local address 127.0.0.1:47418)
0927 15:18:46.814534 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:46.814680 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 15:18:46.815834 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:46.815458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39831 (local address 127.0.0.1:47426)
0927 15:18:46.815624 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:46.815695 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 15:18:46.816016 359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.125:39831: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39831: connect: Connection refused (error 111)
I20260927 15:18:46.907264 359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } attempt: 3
I20260927 15:18:46.908344 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:46.907905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39831 (local address 127.0.0.1:47432)
0927 15:18:46.908121 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:46.908211 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 15:18:46.909304 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:46.908937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39831 (local address 127.0.0.1:47446)
0927 15:18:46.909075 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:46.909149 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 15:18:46.909519 359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.125:39831: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39831: connect: Connection refused (error 111)
W20260927 15:18:47.040989 366 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:18:47.041363 366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:47.041467 366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:47.041584 366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:47.048930 366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:47.049036 366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:47.049116 366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:47.049188 366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:47.049273 366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:47.052750 359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } attempt: 4
I20260927 15:18:47.053979 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.053373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39831 (local address 127.0.0.1:47460)
0927 15:18:47.053675 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.053784 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260927 15:18:47.055022 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.054640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39831 (local address 127.0.0.1:47462)
0927 15:18:47.054779 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.054876 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 15:18:47.055204 359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.125:39831: Network error: Client connection negotiation failed: client connection to 127.10.33.125:39831: connect: Connection refused (error 111)
I20260927 15:18:47.060098 366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:33409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:46507,127.10.33.125:39831,127.10.33.124:41433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.125.keytab
--principal=kudu/127.10.33.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.125:39831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.33.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:47.061386 366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:47.062719 366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:47.074585 371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:47.074944 374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:47.074976 372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:47.075816 366 server_base.cc:1034] running on GCE node
I20260927 15:18:47.076589 366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:47.078045 366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:47.079262 366 hybrid_clock.cc:648] HybridClock initialized: now 1790522327079197 us; error 44 us; skew 500 ppm
I20260927 15:18:47.079567 366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:47 dist-test-slave-f5h6 krb5kdc[32747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522327, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:47.082651 366 init.cc:377] Logged in from keytab as kudu/127.10.33.125@KRBTEST.COM (short username kudu)
I20260927 15:18:47.084115 366 webserver.cc:460] Webserver started at http://127.10.33.125:45005/ using document root <none> and password file <none>
I20260927 15:18:47.084817 366 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:47.084915 366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:47.085191 366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:47.088891 366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-1/data/instance:
uuid: "8bc9410ebbb649cd9bdb107ec62da10c"
format_stamp: "Formatted at 2026-09-27 15:18:47 on dist-test-slave-f5h6"
server_key: "e5427f138a04db470949584f9ac599a1"
server_key_iv: "23043dccf33d5186e9ec4d7bbc8e38b0"
server_key_version: "encryptionkey@0"
I20260927 15:18:47.089787 366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-1/wal/instance:
uuid: "8bc9410ebbb649cd9bdb107ec62da10c"
format_stamp: "Formatted at 2026-09-27 15:18:47 on dist-test-slave-f5h6"
server_key: "e5427f138a04db470949584f9ac599a1"
server_key_iv: "23043dccf33d5186e9ec4d7bbc8e38b0"
server_key_version: "encryptionkey@0"
I20260927 15:18:47.095482 366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:18:47.099597 381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:47.101233 366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:18:47.101438 366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-1/wal
uuid: "8bc9410ebbb649cd9bdb107ec62da10c"
format_stamp: "Formatted at 2026-09-27 15:18:47 on dist-test-slave-f5h6"
server_key: "e5427f138a04db470949584f9ac599a1"
server_key_iv: "23043dccf33d5186e9ec4d7bbc8e38b0"
server_key_version: "encryptionkey@0"
I20260927 15:18:47.101717 366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:47.127352 366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:47.132460 366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:47.132753 366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:47.150931 366 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.125:39831
I20260927 15:18:47.150941 432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.125:39831 every 8 connection(s)
I20260927 15:18:47.152424 366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-1/data/info.pb
I20260927 15:18:47.156288 433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:47.158772 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 366
I20260927 15:18:47.158939 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-1/wal/instance
I20260927 15:18:47.159351 10373 external_mini_cluster.cc:1428] Setting key cf685539a02ef16d23637265b0efb38b
I20260927 15:18:47.163153 433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46507 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:18:47 dist-test-slave-f5h6 krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522327, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.125@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.33.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.124@KRBTEST.COM" created.
I20260927 15:18:47.186815 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.165994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54492 (local address 127.10.33.126:46507)
0927 15:18:47.166224 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:47.166231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:47.167784 (+ 1553us) server_negotiation.cc:407] Connection header received
0927 15:18:47.169791 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:47.169797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:47.169918 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:47.170035 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:47.171661 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.172760 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.174545 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.175131 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.175287 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:47.179216 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:47.179224 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:47.179229 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:47.179278 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:47.182959 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.183840 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.183845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.183847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.183981 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.184466 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.184470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.184472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.184826 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:47.184998 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:47.186105 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0927 15:18:47.186485 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 15:18:47.187829 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.166652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:46507 (local address 127.0.0.1:54492)
0927 15:18:47.167556 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.167611 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:47.169476 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:47.170262 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:47.170271 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:47.170839 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:47.171460 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:47.171480 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.172979 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.172984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:47.174373 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:47.174389 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.175357 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.175362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:47.175586 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:47.176362 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:47.176389 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:47.178898 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:47.183143 (+ 4245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:47.183151 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:47.183170 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:47.183688 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:47.184088 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:47.184091 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:47.184094 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:47.184313 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:47.184996 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:47.185006 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:47.185571 (+ 565us) client_negotiation.cc:769] Sending connection context
0927 15:18:47.185920 (+ 349us) client_negotiation.cc:240] Negotiation successful
0927 15:18:47.186655 (+ 735us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":351,"threads_started":1}
I20260927 15:18:47.189797 433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:18:47 dist-test-slave-f5h6 krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522327, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.125@KRBTEST.COM for kudu/127.10.33.125@KRBTEST.COM
I20260927 15:18:47.205720 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.190505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39831 (local address 127.0.0.1:47466)
0927 15:18:47.190652 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.190671 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:47.190821 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:47.191897 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:47.191900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:47.192259 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:47.192400 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:47.192408 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.194171 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.194175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:47.195093 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:47.195108 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.196285 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.196291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:47.196479 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:47.197219 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:47.197243 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:47.199918 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:47.203296 (+ 3378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:47.203302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:47.203306 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:47.203663 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:47.204022 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:47.204027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:47.204031 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:47.204174 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:47.204770 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:47.204774 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:47.204911 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 15:18:47.205026 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 15:18:47.205308 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 15:18:47.205880 437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.190756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47466 (local address 127.10.33.125:39831)
0927 15:18:47.191375 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:47.191383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:47.191415 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:18:47.191470 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:47.191478 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:47.191723 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:47.191890 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:47.192565 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.193998 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.195296 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.196148 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.196263 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:47.200156 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:47.200171 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:47.200186 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:47.200223 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:47.203157 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.203785 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.203794 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.203797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.203923 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.204283 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.204288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.204291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.204642 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:47.204844 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:47.205205 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 15:18:47.205464 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":280,"threads_started":1}
I20260927 15:18:47.208761 433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:18:47.210045 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.209498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41433 (local address 127.0.0.1:47492)
0927 15:18:47.209695 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.209833 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 15:18:47.210358 384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
I20260927 15:18:47.212029 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.211614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41433 (local address 127.0.0.1:47504)
0927 15:18:47.211783 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.211892 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 15:18:47.212270 433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41433: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.124.keytab.
Entry for principal kudu/127.10.33.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.124.keytab.
I20260927 15:18:47.223096 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.124:41433
--webserver_interface=127.10.33.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.124.keytab
--principal=kudu/127.10.33.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:33409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:46507,127.10.33.125:39831,127.10.33.124:41433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:18:47.261626 433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } attempt: 1
I20260927 15:18:47.262779 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.262320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41433 (local address 127.0.0.1:47518)
0927 15:18:47.262521 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.262618 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 15:18:47.263948 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.263545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41433 (local address 127.0.0.1:47520)
0927 15:18:47.263728 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.263833 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260927 15:18:47.264155 433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41433: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
I20260927 15:18:47.332368 433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } attempt: 2
I20260927 15:18:47.333542 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.333074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41433 (local address 127.0.0.1:47524)
0927 15:18:47.333292 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.333384 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 15:18:47.334556 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.334230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41433 (local address 127.0.0.1:47536)
0927 15:18:47.334378 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.334444 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 15:18:47.334756 433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41433: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
I20260927 15:18:47.354434 359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } attempt: 5
Sep 27 15:18:47 dist-test-slave-f5h6 krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522326, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for kudu/127.10.33.125@KRBTEST.COM
I20260927 15:18:47.367362 437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.355269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47478 (local address 127.10.33.125:39831)
0927 15:18:47.355457 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:47.355463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:47.355499 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:18:47.355557 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:47.355561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:47.355635 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:47.355714 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:47.356548 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.357235 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.358440 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.358879 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.358985 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:47.362321 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:47.362326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:47.362329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:47.362365 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:47.364836 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.365468 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.365473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.365475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.365595 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.365974 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.365979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.365983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.366281 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:47.366412 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:47.366794 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 15:18:47.367036 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:18:47.367358 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.355096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39831 (local address 127.0.0.1:47478)
0927 15:18:47.355273 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.355294 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:47.355429 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:47.355816 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:47.355822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:47.356191 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:47.356389 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:47.356398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.357395 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.357400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:47.358300 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:47.358312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.359019 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.359025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:47.359204 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:47.359780 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:47.359799 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:47.362117 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:47.364970 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:47.364975 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:47.364980 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:47.365354 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:47.365698 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:47.365704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:47.365708 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:47.365851 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:47.366393 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:47.366397 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:47.366556 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 15:18:47.366671 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 15:18:47.366981 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 15:18:47.368675 359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:18:47.369732 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.369365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41433 (local address 127.0.0.1:47540)
0927 15:18:47.369524 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.369598 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 15:18:47.370646 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.370337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41433 (local address 127.0.0.1:47544)
0927 15:18:47.370486 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.370549 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 15:18:47.370823 359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41433: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
I20260927 15:18:47.422075 359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } attempt: 1
I20260927 15:18:47.423431 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.422912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41433 (local address 127.0.0.1:47556)
0927 15:18:47.423164 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.423262 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 15:18:47.424508 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.424158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41433 (local address 127.0.0.1:47562)
0927 15:18:47.424315 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.424390 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 15:18:47.424742 359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41433: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
I20260927 15:18:47.425956 433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } attempt: 3
I20260927 15:18:47.426903 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.426475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41433 (local address 127.0.0.1:47568)
0927 15:18:47.426667 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.426747 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 15:18:47.428005 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.427587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41433 (local address 127.0.0.1:47570)
0927 15:18:47.427811 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.427889 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260927 15:18:47.428206 433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41433: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
I20260927 15:18:47.492978 359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } attempt: 2
I20260927 15:18:47.494271 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.493713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41433 (local address 127.0.0.1:47574)
0927 15:18:47.493974 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.494067 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 15:18:47.495234 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.494895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41433 (local address 127.0.0.1:47578)
0927 15:18:47.495050 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.495122 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 15:18:47.495438 359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41433: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
W20260927 15:18:47.567013 440 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:18:47.567394 440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:47.567490 440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:47.567600 440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 15:18:47.571460 433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } attempt: 4
I20260927 15:18:47.572558 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.572065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41433 (local address 127.0.0.1:47594)
0927 15:18:47.572313 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.572399 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 15:18:47.573529 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.573214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41433 (local address 127.0.0.1:47608)
0927 15:18:47.573362 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.573427 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 15:18:47.573767 433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41433: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
W20260927 15:18:47.575353 440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:47.575453 440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:47.575543 440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:47.575608 440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:47.575690 440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:47.585986 440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:33409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:46507,127.10.33.125:39831,127.10.33.124:41433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.124.keytab
--principal=kudu/127.10.33.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.124:41433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.33.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:47.587340 440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:47.588639 440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:47.600284 445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:47.600523 448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:47.600523 446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:47.600919 440 server_base.cc:1034] running on GCE node
I20260927 15:18:47.601773 440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:47.601709 359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } attempt: 3
I20260927 15:18:47.602885 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.602336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41433 (local address 127.0.0.1:47622)
0927 15:18:47.602574 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.602677 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 15:18:47.603612 440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:47.603935 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.603558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41433 (local address 127.0.0.1:47636)
0927 15:18:47.603752 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.603831 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260927 15:18:47.604122 359 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41433: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41433: connect: Connection refused (error 111)
I20260927 15:18:47.604861 440 hybrid_clock.cc:648] HybridClock initialized: now 1790522327604786 us; error 49 us; skew 500 ppm
I20260927 15:18:47.605199 440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:47 dist-test-slave-f5h6 krb5kdc[32747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522327, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:47.608796 440 init.cc:377] Logged in from keytab as kudu/127.10.33.124@KRBTEST.COM (short username kudu)
I20260927 15:18:47.610432 440 webserver.cc:460] Webserver started at http://127.10.33.124:44365/ using document root <none> and password file <none>
I20260927 15:18:47.611187 440 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:47.611300 440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:47.611609 440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:47.614972 440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-2/data/instance:
uuid: "1947c28b09bc4dc0bf9cafee37e26c82"
format_stamp: "Formatted at 2026-09-27 15:18:47 on dist-test-slave-f5h6"
server_key: "ddb6c92b737e01f92abdac7e8bc23913"
server_key_iv: "b9a12dadfb9e5e4b1febb292daf4a763"
server_key_version: "encryptionkey@0"
I20260927 15:18:47.615835 440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-2/wal/instance:
uuid: "1947c28b09bc4dc0bf9cafee37e26c82"
format_stamp: "Formatted at 2026-09-27 15:18:47 on dist-test-slave-f5h6"
server_key: "ddb6c92b737e01f92abdac7e8bc23913"
server_key_iv: "b9a12dadfb9e5e4b1febb292daf4a763"
server_key_version: "encryptionkey@0"
I20260927 15:18:47.621371 440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:18:47.625232 455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:47.626971 440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:18:47.627170 440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-2/wal
uuid: "1947c28b09bc4dc0bf9cafee37e26c82"
format_stamp: "Formatted at 2026-09-27 15:18:47 on dist-test-slave-f5h6"
server_key: "ddb6c92b737e01f92abdac7e8bc23913"
server_key_iv: "b9a12dadfb9e5e4b1febb292daf4a763"
server_key_version: "encryptionkey@0"
I20260927 15:18:47.627408 440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:47.650889 440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:47.655490 440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:47.655762 440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:47.673774 440 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.124:41433
I20260927 15:18:47.673806 506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.124:41433 every 8 connection(s)
I20260927 15:18:47.675184 440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-2/data/info.pb
I20260927 15:18:47.678069 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 440
I20260927 15:18:47.678223 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521968392480-10373-0/minicluster-data/master-2/wal/instance
I20260927 15:18:47.678607 10373 external_mini_cluster.cc:1428] Setting key f79ce30159542bd300978654a1e81339
I20260927 15:18:47.679080 507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:47.687889 507 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46507 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:18:47 dist-test-slave-f5h6 krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.124@KRBTEST.COM
Sep 27 15:18:47 dist-test-slave-f5h6 krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522327, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.124@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:18:47.701992 510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.680349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47638 (local address 127.10.33.124:41433)
0927 15:18:47.681085 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:47.681097 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:47.681135 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:18:47.683570 (+ 2435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:47.683595 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:47.684017 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:47.684436 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:47.686003 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.687376 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.688670 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.689414 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.689574 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:47.694185 (+ 4611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:47.694193 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:47.694205 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:47.694247 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:47.697564 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.698342 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.698354 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.698361 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.698522 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.698898 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.698901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.698903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.699631 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:47.699925 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:47.700422 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 15:18:47.700840 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":331,"threads_started":1}
I20260927 15:18:47.705996 512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.690655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:46507 (local address 127.0.0.1:54498)
0927 15:18:47.691423 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.691458 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:47.691632 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:47.692012 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:47.692022 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:47.692768 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:47.693110 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:47.693125 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.694272 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.694276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:47.695568 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:47.695585 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.696338 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.696345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:47.696499 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:47.697246 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:47.697272 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:47.699729 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:47.703273 (+ 3544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:47.703285 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:47.703292 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:47.703779 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:47.704159 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:47.704164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:47.704167 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:47.704346 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:47.704905 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:47.704913 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:47.705105 (+ 192us) client_negotiation.cc:769] Sending connection context
0927 15:18:47.705303 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 15:18:47.705658 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":423,"threads_started":1}
I20260927 15:18:47.706410 511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.690603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54498 (local address 127.10.33.126:46507)
0927 15:18:47.691086 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:47.691093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:47.691564 (+ 471us) server_negotiation.cc:407] Connection header received
0927 15:18:47.691761 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:47.691766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:47.691832 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:47.691922 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:47.693272 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.694105 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.695724 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.696133 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.696251 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:47.699956 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:47.699965 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:47.699970 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:47.700019 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:47.703107 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.703908 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.703913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.703915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.704049 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.704445 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.704450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.704453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.704758 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:47.704906 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:47.705486 (+ 580us) server_negotiation.cc:299] Negotiation successful
0927 15:18:47.705972 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":231,"threads_started":1}
I20260927 15:18:47.708066 507 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:18:47 dist-test-slave-f5h6 krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522327, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.124@KRBTEST.COM for kudu/127.10.33.125@KRBTEST.COM
Sep 27 15:18:47 dist-test-slave-f5h6 krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:18:47.723867 511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.705121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54514 (local address 127.10.33.126:46507)
0927 15:18:47.706694 (+ 1573us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:47.706700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:47.706720 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:18:47.706795 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:47.706801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:47.706906 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:47.707014 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:47.709046 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.710114 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.712123 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.712788 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.712902 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:47.717977 (+ 5075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:47.717983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:47.717986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:47.718022 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:47.721020 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.721807 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.721811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.721814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.721989 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.722387 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.722392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.722396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.722702 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:47.722960 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:47.723282 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 15:18:47.723515 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1456,"thread_start_us":15581,"threads_started":1}
I20260927 15:18:47.724512 512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.708982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:39831 (local address 127.0.0.1:47482)
0927 15:18:47.709337 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.709358 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:47.709475 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:47.709813 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:47.709818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:47.710308 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:47.710453 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:47.710461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.711399 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.711404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:47.712540 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:47.712556 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.713639 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.713646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:47.713823 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:47.714381 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:47.714407 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:47.717078 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:47.721349 (+ 4271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:47.721355 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:47.721359 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:47.722106 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:47.722543 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:47.722548 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:47.722552 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:47.722770 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:47.723676 (+ 906us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:47.723681 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:47.723856 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 15:18:47.723985 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 15:18:47.724289 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214}
I20260927 15:18:47.724743 437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.709096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47482 (local address 127.10.33.125:39831)
0927 15:18:47.709328 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:47.709336 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:47.709411 (+ 75us) server_negotiation.cc:407] Connection header received
0927 15:18:47.709587 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:47.709593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:47.709674 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:47.709786 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:47.710597 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.711260 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.712716 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.713454 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.713588 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:47.717329 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:47.717337 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:47.717343 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:47.717393 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:47.721074 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.722262 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.722268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.722270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.722419 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.722896 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.722900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.722903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.723528 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:47.723738 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:47.724155 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 15:18:47.724398 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 15:18:47.725574 507 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:18:47 dist-test-slave-f5h6 krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522327, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.124@KRBTEST.COM for kudu/127.10.33.124@KRBTEST.COM
Sep 27 15:18:47 dist-test-slave-f5h6 krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.125@KRBTEST.COM
I20260927 15:18:47.738596 512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.726326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41433 (local address 127.0.0.1:47648)
0927 15:18:47.726530 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.726543 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:47.726644 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:47.726920 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:47.726924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:47.727308 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:47.727441 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:47.727449 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.728342 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.728347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:47.729192 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:47.729204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.729901 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.729907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:47.730080 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:47.730725 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:47.730744 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:47.733074 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:47.736215 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:47.736221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:47.736225 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:47.736582 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:47.736925 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:47.736930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:47.736934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:47.737061 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:47.737559 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:47.737564 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:47.737704 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 15:18:47.737818 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 15:18:47.738184 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":4}
I20260927 15:18:47.738636 510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.726326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47648 (local address 127.10.33.124:41433)
0927 15:18:47.726463 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:47.726467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:47.726606 (+ 139us) server_negotiation.cc:407] Connection header received
0927 15:18:47.726756 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:47.726761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:47.726819 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:47.726892 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:47.727561 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.728230 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.729326 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.729733 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.729838 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:47.733253 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:47.733258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:47.733269 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:47.733299 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:47.736060 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.736703 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.736708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.736712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.736818 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.737157 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.737163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.737166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.737442 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:47.737577 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:47.737930 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 15:18:47.738344 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":175,"server-negotiator.queue_time_us":46}
I20260927 15:18:47.741372 437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.726076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47488 (local address 127.10.33.125:39831)
0927 15:18:47.726247 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:47.726253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:47.726639 (+ 386us) server_negotiation.cc:407] Connection header received
0927 15:18:47.726788 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:47.726793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:47.726855 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:47.726929 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:47.728155 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.728986 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.730257 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.730811 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.730923 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:47.735599 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:47.735605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:47.735608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:47.735647 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:47.738229 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.738991 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.738997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.739000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.739237 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.739734 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.739740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.739745 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.740114 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:47.740495 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:47.740836 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 15:18:47.741077 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:18:47.744654 507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1947c28b09bc4dc0bf9cafee37e26c82: Bootstrap starting.
I20260927 15:18:47.746918 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:18:47.748858 507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1947c28b09bc4dc0bf9cafee37e26c82: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:47.750520 507 log.cc:824] T 00000000000000000000000000000000 P 1947c28b09bc4dc0bf9cafee37e26c82: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:47.753437 507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1947c28b09bc4dc0bf9cafee37e26c82: No bootstrap required, opened a new log
I20260927 15:18:47.750545 334 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "64541b67c04e43dd93642309f5643a4d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54514
I20260927 15:18:47.760617 507 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1947c28b09bc4dc0bf9cafee37e26c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64541b67c04e43dd93642309f5643a4d" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46507 } } peers { permanent_uuid: "8bc9410ebbb649cd9bdb107ec62da10c" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } } peers { permanent_uuid: "1947c28b09bc4dc0bf9cafee37e26c82" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } }
I20260927 15:18:47.760991 507 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1947c28b09bc4dc0bf9cafee37e26c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1947c28b09bc4dc0bf9cafee37e26c82, State: Initialized, Role: FOLLOWER
I20260927 15:18:47.761417 507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1947c28b09bc4dc0bf9cafee37e26c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64541b67c04e43dd93642309f5643a4d" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46507 } } peers { permanent_uuid: "8bc9410ebbb649cd9bdb107ec62da10c" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } } peers { permanent_uuid: "1947c28b09bc4dc0bf9cafee37e26c82" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } }
I20260927 15:18:47.762336 515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1947c28b09bc4dc0bf9cafee37e26c82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64541b67c04e43dd93642309f5643a4d" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46507 } } peers { permanent_uuid: "8bc9410ebbb649cd9bdb107ec62da10c" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } } peers { permanent_uuid: "1947c28b09bc4dc0bf9cafee37e26c82" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } } }
I20260927 15:18:47.762653 515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1947c28b09bc4dc0bf9cafee37e26c82 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:18:47.763592 507 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1947c28b09bc4dc0bf9cafee37e26c82 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 15:18:47.773574 526 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1947c28b09bc4dc0bf9cafee37e26c82: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:18:47.773783 526 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:18:47.781342 359 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } attempt: 4
I20260927 15:18:47.771502 440 master_runner.cc:186] Error getting master registration for 127.10.33.126:46507: OK, instance_id { permanent_uuid: "64541b67c04e43dd93642309f5643a4d" instance_seqno: 1790522326642207 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:18:47.782544 440 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 15:18:47 dist-test-slave-f5h6 krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522326, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for kudu/127.10.33.124@KRBTEST.COM
I20260927 15:18:47.798295 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.782090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41433 (local address 127.0.0.1:47656)
0927 15:18:47.782602 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.782621 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:47.782772 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:47.783139 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:47.783143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:47.783650 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:47.783863 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:47.783876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.785339 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.785344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:47.786828 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:47.786844 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.787861 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.787867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:47.788073 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:47.788775 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:47.788798 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:47.791718 (+ 2920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:47.795345 (+ 3627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:47.795351 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:47.795356 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:47.795930 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:47.796432 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:47.796438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:47.796442 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:47.796586 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:47.797236 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:47.797241 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:47.797409 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 15:18:47.797529 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 15:18:47.797837 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387}
I20260927 15:18:47.798281 510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.782179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47656 (local address 127.10.33.124:41433)
0927 15:18:47.782439 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:47.782454 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:47.782700 (+ 246us) server_negotiation.cc:407] Connection header received
0927 15:18:47.782902 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:47.782906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:47.782988 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:47.783107 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:47.784026 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.785182 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.786981 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.787709 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.787856 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:47.791945 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:47.791951 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:47.791954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:47.791991 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:47.795170 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.796096 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.796100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.796103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.796282 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.796757 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.796761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.796763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.797104 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:47.797259 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:47.797668 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 15:18:47.797910 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 15:18:47.803165 359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d: Bootstrap starting.
I20260927 15:18:47.806919 359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:47.808084 359 log.cc:824] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:47.811203 359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d: No bootstrap required, opened a new log
I20260927 15:18:47.816448 359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64541b67c04e43dd93642309f5643a4d" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46507 } } peers { permanent_uuid: "8bc9410ebbb649cd9bdb107ec62da10c" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } } peers { permanent_uuid: "1947c28b09bc4dc0bf9cafee37e26c82" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } }
I20260927 15:18:47.816789 359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64541b67c04e43dd93642309f5643a4d, State: Initialized, Role: FOLLOWER
I20260927 15:18:47.817231 359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64541b67c04e43dd93642309f5643a4d" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46507 } } peers { permanent_uuid: "8bc9410ebbb649cd9bdb107ec62da10c" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } } peers { permanent_uuid: "1947c28b09bc4dc0bf9cafee37e26c82" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } }
I20260927 15:18:47.818198 529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64541b67c04e43dd93642309f5643a4d" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46507 } } peers { permanent_uuid: "8bc9410ebbb649cd9bdb107ec62da10c" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } } peers { permanent_uuid: "1947c28b09bc4dc0bf9cafee37e26c82" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } } }
I20260927 15:18:47.818617 529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:18:47.819473 359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:47.819550 334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 15:18:47.819751 334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:47.823628 334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64541b67c04e43dd93642309f5643a4d" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46507 } } peers { permanent_uuid: "8bc9410ebbb649cd9bdb107ec62da10c" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } } peers { permanent_uuid: "1947c28b09bc4dc0bf9cafee37e26c82" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } }
I20260927 15:18:47.826965 482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "64541b67c04e43dd93642309f5643a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1947c28b09bc4dc0bf9cafee37e26c82"
I20260927 15:18:47.827625 482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1947c28b09bc4dc0bf9cafee37e26c82 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:47.832398 482 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1947c28b09bc4dc0bf9cafee37e26c82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64541b67c04e43dd93642309f5643a4d in term 1.
I20260927 15:18:47.833581 310 leader_election.cc:302] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1947c28b09bc4dc0bf9cafee37e26c82, 64541b67c04e43dd93642309f5643a4d; no voters:
I20260927 15:18:47.834286 529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:47.834376 334 leader_election.cc:288] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d [CANDIDATE]: Term 1 election: Requested vote from peers 8bc9410ebbb649cd9bdb107ec62da10c (127.10.33.125:39831), 1947c28b09bc4dc0bf9cafee37e26c82 (127.10.33.124:41433)
I20260927 15:18:47.847200 529 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d [term 1 LEADER]: Becoming Leader. State: Replica: 64541b67c04e43dd93642309f5643a4d, State: Running, Role: LEADER
I20260927 15:18:47.848585 529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64541b67c04e43dd93642309f5643a4d" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46507 } } peers { permanent_uuid: "8bc9410ebbb649cd9bdb107ec62da10c" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } } peers { permanent_uuid: "1947c28b09bc4dc0bf9cafee37e26c82" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } }
I20260927 15:18:47.863183 533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64541b67c04e43dd93642309f5643a4d. Latest consensus state: current_term: 1 leader_uuid: "64541b67c04e43dd93642309f5643a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64541b67c04e43dd93642309f5643a4d" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46507 } } peers { permanent_uuid: "8bc9410ebbb649cd9bdb107ec62da10c" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } } peers { permanent_uuid: "1947c28b09bc4dc0bf9cafee37e26c82" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } } }
I20260927 15:18:47.863680 533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:47.872938 433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } attempt: 5
W20260927 15:18:47.882654 313 tablet.cc:2369] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:18:47.895288 543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260927 15:18:47.907918 461 tablet.cc:2369] T 00000000000000000000000000000000 P 1947c28b09bc4dc0bf9cafee37e26c82: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 15:18:47 dist-test-slave-f5h6 krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522327, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.125@KRBTEST.COM for kudu/127.10.33.124@KRBTEST.COM
I20260927 15:18:47.920146 511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.863393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54526 (local address 127.10.33.126:46507)
0927 15:18:47.863615 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:47.863623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:47.884678 (+ 21055us) server_negotiation.cc:407] Connection header received
0927 15:18:47.886083 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:47.886091 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:47.886201 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:47.886296 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:47.893200 (+ 6904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.894494 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.900964 (+ 6470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.901724 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.901830 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:47.911827 (+ 9997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:47.911836 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:47.911841 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:47.911895 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:47.915989 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.917031 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.917039 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.917043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.917274 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.917791 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.917798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.917801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.918314 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:47.918604 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:47.919298 (+ 694us) server_negotiation.cc:299] Negotiation successful
0927 15:18:47.919612 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:18:47.929129 543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:47.935159 510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.869809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47660 (local address 127.10.33.124:41433)
0927 15:18:47.870020 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:47.870027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:47.885929 (+ 15902us) server_negotiation.cc:407] Connection header received
0927 15:18:47.889364 (+ 3435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:47.889372 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:47.889472 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:47.889551 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:47.896754 (+ 7203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.898136 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.903138 (+ 5002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.903875 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.903983 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:47.923036 (+ 19053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:47.923046 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:47.923051 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:47.923111 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:47.928776 (+ 5665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.930000 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.930007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.930011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.930197 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.931727 (+ 1530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.931734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.931737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.932568 (+ 831us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:47.933013 (+ 445us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:47.934016 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0927 15:18:47.934447 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 15:18:47.935487 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.873564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41433 (local address 127.0.0.1:47674)
0927 15:18:47.873848 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:47.873891 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:47.874010 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:47.897740 (+ 23730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:47.897747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:47.898346 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:47.898583 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:47.898597 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.903790 (+ 5193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.903796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:47.905342 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:47.905359 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.906458 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.906465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:47.906726 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:47.909744 (+ 3018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:47.909769 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:47.912715 (+ 2946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:47.932420 (+ 19705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:47.932427 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:47.932431 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:47.933010 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:47.933481 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:47.933486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:47.933489 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:47.933663 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:47.934433 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:47.934438 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:47.934597 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 15:18:47.934732 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 15:18:47.935072 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260927 15:18:47.936774 546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.874489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47674 (local address 127.10.33.124:41433)
0927 15:18:47.894367 (+ 19878us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:47.894373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:47.894401 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:18:47.894460 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:47.894465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:47.894561 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:47.898817 (+ 4256us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:47.899107 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.900362 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.905545 (+ 5183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.906267 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.906394 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:47.915151 (+ 8757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:47.915160 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:47.915164 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:47.915217 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:47.932197 (+ 16980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.933167 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.933171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.933174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.933335 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.933812 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.933816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.933819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.934268 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:47.935729 (+ 1461us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:47.935880 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 15:18:47.936096 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19727,"spinlock_wait_cycles":23296,"thread_start_us":20197,"threads_started":1}
I20260927 15:18:47.943495 433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc9410ebbb649cd9bdb107ec62da10c: Bootstrap starting.
I20260927 15:18:47.949401 433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bc9410ebbb649cd9bdb107ec62da10c: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:47.951256 433 log.cc:824] T 00000000000000000000000000000000 P 8bc9410ebbb649cd9bdb107ec62da10c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:47.955374 433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc9410ebbb649cd9bdb107ec62da10c: No bootstrap required, opened a new log
I20260927 15:18:47.959515 437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:47.892574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.10.33.125:39831)
0927 15:18:47.892759 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:47.892765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:47.901072 (+ 8307us) server_negotiation.cc:407] Connection header received
0927 15:18:47.901325 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:47.901331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:47.901412 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:47.901477 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:47.908022 (+ 6545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.909507 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.930003 (+ 20496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:47.930746 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:47.930889 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:47.951310 (+ 20421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:47.951317 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:47.951323 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:47.951377 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:47.955913 (+ 4536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.957032 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.957038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.957042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.957233 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:47.957750 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:47.957755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:47.957759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:47.958221 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:47.958506 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:47.958862 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 15:18:47.959137 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:18:47.967528 433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bc9410ebbb649cd9bdb107ec62da10c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64541b67c04e43dd93642309f5643a4d" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46507 } } peers { permanent_uuid: "8bc9410ebbb649cd9bdb107ec62da10c" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } } peers { permanent_uuid: "1947c28b09bc4dc0bf9cafee37e26c82" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } }
I20260927 15:18:47.967988 433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bc9410ebbb649cd9bdb107ec62da10c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc9410ebbb649cd9bdb107ec62da10c, State: Initialized, Role: FOLLOWER
I20260927 15:18:47.968566 433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bc9410ebbb649cd9bdb107ec62da10c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64541b67c04e43dd93642309f5643a4d" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46507 } } peers { permanent_uuid: "8bc9410ebbb649cd9bdb107ec62da10c" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } } peers { permanent_uuid: "1947c28b09bc4dc0bf9cafee37e26c82" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } }
I20260927 15:18:47.970103 554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc9410ebbb649cd9bdb107ec62da10c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64541b67c04e43dd93642309f5643a4d" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46507 } } peers { permanent_uuid: "8bc9410ebbb649cd9bdb107ec62da10c" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } } peers { permanent_uuid: "1947c28b09bc4dc0bf9cafee37e26c82" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } } }
I20260927 15:18:47.970908 554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc9410ebbb649cd9bdb107ec62da10c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:18:47.971449 433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bc9410ebbb649cd9bdb107ec62da10c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:47.936156 32760 master_runner.cc:186] Error getting master registration for 127.10.33.125:39831: OK, instance_id { permanent_uuid: "8bc9410ebbb649cd9bdb107ec62da10c" instance_seqno: 1790522327145592 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:18:47.971493 408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "64541b67c04e43dd93642309f5643a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8bc9410ebbb649cd9bdb107ec62da10c"
I20260927 15:18:47.971977 408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bc9410ebbb649cd9bdb107ec62da10c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:47.975976 32760 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:18:47.975965 408 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8bc9410ebbb649cd9bdb107ec62da10c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64541b67c04e43dd93642309f5643a4d in term 1.
I20260927 15:18:47.993079 482 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1947c28b09bc4dc0bf9cafee37e26c82 [term 1 FOLLOWER]: Refusing update from remote peer 64541b67c04e43dd93642309f5643a4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:18:47.994294 533 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1947c28b09bc4dc0bf9cafee37e26c82" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:18:47.997660 408 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8bc9410ebbb649cd9bdb107ec62da10c [term 1 FOLLOWER]: Refusing update from remote peer 64541b67c04e43dd93642309f5643a4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:18:47.999455 533 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bc9410ebbb649cd9bdb107ec62da10c" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:18:48.014045 566 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8bc9410ebbb649cd9bdb107ec62da10c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:18:48.016404 566 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:18:48.018491 515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1947c28b09bc4dc0bf9cafee37e26c82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64541b67c04e43dd93642309f5643a4d. Latest consensus state: current_term: 1 leader_uuid: "64541b67c04e43dd93642309f5643a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64541b67c04e43dd93642309f5643a4d" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46507 } } peers { permanent_uuid: "8bc9410ebbb649cd9bdb107ec62da10c" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } } peers { permanent_uuid: "1947c28b09bc4dc0bf9cafee37e26c82" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } } }
I20260927 15:18:48.018909 515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1947c28b09bc4dc0bf9cafee37e26c82 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:18:48.021757 554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc9410ebbb649cd9bdb107ec62da10c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64541b67c04e43dd93642309f5643a4d. Latest consensus state: current_term: 1 leader_uuid: "64541b67c04e43dd93642309f5643a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64541b67c04e43dd93642309f5643a4d" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46507 } } peers { permanent_uuid: "8bc9410ebbb649cd9bdb107ec62da10c" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } } peers { permanent_uuid: "1947c28b09bc4dc0bf9cafee37e26c82" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } } }
I20260927 15:18:48.022116 554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc9410ebbb649cd9bdb107ec62da10c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:18:48.025727 555 mvcc.cc:205] Tried to move back new op lower bound from 7333979455432806400 to 7333979454879789056. Current Snapshot: MvccSnapshot[applied={T|T < 7333979455432806400}]
I20260927 15:18:48.026481 529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64541b67c04e43dd93642309f5643a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64541b67c04e43dd93642309f5643a4d" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46507 } } peers { permanent_uuid: "8bc9410ebbb649cd9bdb107ec62da10c" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } } peers { permanent_uuid: "1947c28b09bc4dc0bf9cafee37e26c82" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } } }
I20260927 15:18:48.026830 529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:48.027243 568 mvcc.cc:205] Tried to move back new op lower bound from 7333979455432806400 to 7333979454879789056. Current Snapshot: MvccSnapshot[applied={T|T < 7333979455432806400}]
I20260927 15:18:48.028707 533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64541b67c04e43dd93642309f5643a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64541b67c04e43dd93642309f5643a4d" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46507 } } peers { permanent_uuid: "8bc9410ebbb649cd9bdb107ec62da10c" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } } peers { permanent_uuid: "1947c28b09bc4dc0bf9cafee37e26c82" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } } }
I20260927 15:18:48.028998 533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:48.032974 569 mvcc.cc:205] Tried to move back new op lower bound from 7333979455432806400 to 7333979454879789056. Current Snapshot: MvccSnapshot[applied={T|T < 7333979455432806400}]
I20260927 15:18:48.034483 543 catalog_manager.cc:1347] Generated new cluster ID: 8e0fd6b05d3646c7aedda13ff35d6463
I20260927 15:18:48.034612 543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:48.038371 554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc9410ebbb649cd9bdb107ec62da10c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64541b67c04e43dd93642309f5643a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64541b67c04e43dd93642309f5643a4d" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46507 } } peers { permanent_uuid: "8bc9410ebbb649cd9bdb107ec62da10c" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } } peers { permanent_uuid: "1947c28b09bc4dc0bf9cafee37e26c82" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } } }
I20260927 15:18:48.038703 554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc9410ebbb649cd9bdb107ec62da10c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:18:48.040342 515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1947c28b09bc4dc0bf9cafee37e26c82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64541b67c04e43dd93642309f5643a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64541b67c04e43dd93642309f5643a4d" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 46507 } } peers { permanent_uuid: "8bc9410ebbb649cd9bdb107ec62da10c" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 39831 } } peers { permanent_uuid: "1947c28b09bc4dc0bf9cafee37e26c82" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41433 } } }
I20260927 15:18:48.040606 515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1947c28b09bc4dc0bf9cafee37e26c82 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:18:48.108197 543 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:48.110625 543 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:48.159615 543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64541b67c04e43dd93642309f5643a4d: Generated new TSK 0
I20260927 15:18:48.160960 543 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:18:48.308895 511 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:48.284153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54528 (local address 127.10.33.126:46507)
0927 15:18:48.284374 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:48.284380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:48.306815 (+ 22435us) server_negotiation.cc:407] Connection header received
0927 15:18:48.307124 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:48.307130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:48.307221 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:48.307303 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:48.308619 (+ 1316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54528: BlockingRecv error: recv got EOF from 127.0.0.1:54528 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 15:18:48.311920 546 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:48.291033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47676 (local address 127.10.33.124:41433)
0927 15:18:48.291259 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:48.291267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:48.309900 (+ 18633us) server_negotiation.cc:407] Connection header received
0927 15:18:48.310340 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:48.310347 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:48.310433 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:48.310505 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:48.311656 (+ 1151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47676: BlockingRecv error: recv got EOF from 127.0.0.1:47676 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 15:18:48.315856 437 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:48.313362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47498 (local address 127.10.33.125:39831)
0927 15:18:48.313545 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:48.313551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:48.314197 (+ 646us) server_negotiation.cc:407] Connection header received
0927 15:18:48.314438 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:48.314444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:48.314548 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:48.314627 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:48.315581 (+ 954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47498: BlockingRecv error: recv got EOF from 127.0.0.1:47498 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 15:18:48.361002 437 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:48.347619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47514 (local address 127.10.33.125:39831)
0927 15:18:48.347803 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:48.347809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:48.358531 (+ 10722us) server_negotiation.cc:407] Connection header received
0927 15:18:48.359129 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:48.359136 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:48.359278 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:48.359363 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:48.360679 (+ 1316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47514: BlockingRecv error: recv got EOF from 127.0.0.1:47514 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 15:18:48.390730 437 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:48.387789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47520 (local address 127.10.33.125:39831)
0927 15:18:48.387992 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:48.387999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:48.389041 (+ 1042us) server_negotiation.cc:407] Connection header received
0927 15:18:48.389308 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:48.389315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:48.389420 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:48.389542 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:48.390476 (+ 934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47520: BlockingRecv error: recv got EOF from 127.0.0.1:47520 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 15:18:48.400496 437 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:48.397883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.10.33.125:39831)
0927 15:18:48.398072 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:48.398078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:48.398600 (+ 522us) server_negotiation.cc:407] Connection header received
0927 15:18:48.398907 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:48.398914 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:48.399028 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:48.399100 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:48.400212 (+ 1112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47526: BlockingRecv error: recv got EOF from 127.0.0.1:47526 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 15:18:48.411692 437 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:48.409155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47534 (local address 127.10.33.125:39831)
0927 15:18:48.409366 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:48.409376 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:48.409917 (+ 541us) server_negotiation.cc:407] Connection header received
0927 15:18:48.410225 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:48.410232 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:48.410339 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:48.410416 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:48.411432 (+ 1016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47534: BlockingRecv error: recv got EOF from 127.0.0.1:47534 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 15:18:48.426865 437 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:48.424120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47540 (local address 127.10.33.125:39831)
0927 15:18:48.424361 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:48.424368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:48.424944 (+ 576us) server_negotiation.cc:407] Connection header received
0927 15:18:48.425279 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:48.425286 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:48.425404 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:48.425507 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:48.426606 (+ 1099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47540: BlockingRecv error: recv got EOF from 127.0.0.1:47540 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260927 15:18:48.450484 437 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:48.447714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47556 (local address 127.10.33.125:39831)
0927 15:18:48.447921 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:48.447927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:48.448489 (+ 562us) server_negotiation.cc:407] Connection header received
0927 15:18:48.448813 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:48.448823 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:48.448927 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:48.449020 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:48.450186 (+ 1166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47556: BlockingRecv error: recv got EOF from 127.0.0.1:47556 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 15:18:48.489957 437 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:48.487231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47560 (local address 127.10.33.125:39831)
0927 15:18:48.487444 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:48.487450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:48.488019 (+ 569us) server_negotiation.cc:407] Connection header received
0927 15:18:48.488347 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:48.488354 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:48.488455 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:48.488533 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:48.489618 (+ 1085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47560: BlockingRecv error: recv got EOF from 127.0.0.1:47560 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 15:18:48.560084 437 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:48.558161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47572 (local address 127.10.33.125:39831)
0927 15:18:48.558399 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:48.558405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:48.558696 (+ 291us) server_negotiation.cc:407] Connection header received
0927 15:18:48.558874 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:48.558878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:48.558968 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:48.559060 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:48.559873 (+ 813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47572: BlockingRecv error: recv got EOF from 127.0.0.1:47572 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260927 15:18:48.693668 437 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:48.691776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47584 (local address 127.10.33.125:39831)
0927 15:18:48.692010 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:48.692017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:48.692340 (+ 323us) server_negotiation.cc:407] Connection header received
0927 15:18:48.692518 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:48.692523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:48.692600 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:48.692747 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:48.693445 (+ 698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47584: BlockingRecv error: recv got EOF from 127.0.0.1:47584 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 15:18:48.775858 526 catalog_manager.cc:1259] Loaded cluster ID: 8e0fd6b05d3646c7aedda13ff35d6463
I20260927 15:18:48.775969 526 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1947c28b09bc4dc0bf9cafee37e26c82: loading cluster ID for follower catalog manager: success
I20260927 15:18:48.778939 526 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1947c28b09bc4dc0bf9cafee37e26c82: acquiring CA information for follower catalog manager: success
I20260927 15:18:48.781193 526 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1947c28b09bc4dc0bf9cafee37e26c82: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 15:18:48.956799 437 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:48.954642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.10.33.125:39831)
0927 15:18:48.954886 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:48.954892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:48.955098 (+ 206us) server_negotiation.cc:407] Connection header received
0927 15:18:48.955359 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:48.955365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:48.955467 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:48.955593 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:48.956554 (+ 961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47590: BlockingRecv error: recv got EOF from 127.0.0.1:47590 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:18:49.018620 566 catalog_manager.cc:1259] Loaded cluster ID: 8e0fd6b05d3646c7aedda13ff35d6463
I20260927 15:18:49.018741 566 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8bc9410ebbb649cd9bdb107ec62da10c: loading cluster ID for follower catalog manager: success
I20260927 15:18:49.021335 566 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8bc9410ebbb649cd9bdb107ec62da10c: acquiring CA information for follower catalog manager: success
I20260927 15:18:49.023262 566 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8bc9410ebbb649cd9bdb107ec62da10c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:18:49.479789 639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:49.472697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47602 (local address 127.10.33.125:39831)
0927 15:18:49.473249 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:49.473254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:49.473279 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:18:49.473381 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:49.473385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:49.473589 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:49.473761 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:49.474406 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:49.475255 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:49.476746 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:49.477259 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:49.477378 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:49.478193 (+ 815us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:49.478935 (+ 742us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:49.479088 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:49.479327 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 15:18:49.479434 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":278,"threads_started":1}
I20260927 15:18:49.494119 645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:49.487705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47682 (local address 127.10.33.124:41433)
0927 15:18:49.488285 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:49.488290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:49.488343 (+ 53us) server_negotiation.cc:407] Connection header received
0927 15:18:49.488464 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:49.488469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:49.488661 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:49.488854 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:49.489358 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:49.490190 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:49.491646 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:49.492071 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:49.492182 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:49.492785 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:49.493402 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:49.493535 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:49.493677 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 15:18:49.493758 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":363,"threads_started":1}
I20260927 15:18:49.510959 645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:49.502081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47684 (local address 127.10.33.124:41433)
0927 15:18:49.502283 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:49.502290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:49.503886 (+ 1596us) server_negotiation.cc:407] Connection header received
0927 15:18:49.504247 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:49.504253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:49.504356 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:49.504439 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:49.505337 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:49.506240 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:49.508689 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:49.509268 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:49.509362 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:49.509977 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:49.510191 (+ 214us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:49.510296 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:49.510474 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 15:18:49.510602 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:18:49.511116 651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:49.501914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54544 (local address 127.10.33.126:46507)
0927 15:18:49.502804 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:49.502811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:49.503305 (+ 494us) server_negotiation.cc:407] Connection header received
0927 15:18:49.503924 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:49.503931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:49.504190 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:49.504407 (+ 217us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:49.505097 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:49.506291 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:49.508536 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:49.508993 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:49.509129 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:49.509948 (+ 819us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:49.510456 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:49.510576 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:49.510694 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 15:18:49.510811 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":514,"threads_started":1}
I20260927 15:18:49.512126 639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:49.503257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47614 (local address 127.10.33.125:39831)
0927 15:18:49.503437 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:49.503444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:49.504230 (+ 786us) server_negotiation.cc:407] Connection header received
0927 15:18:49.504371 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:49.504376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:49.504486 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:49.504581 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:49.505510 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:49.506690 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:49.509127 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:49.509767 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:49.509917 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:49.511317 (+ 1400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:49.511467 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:49.511552 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:49.511701 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 15:18:49.511799 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:18:49.518401 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 32760
I20260927 15:18:49.519740 354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:49.647848 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 32760
W20260927 15:18:49.666534 457 connection.cc:535] server connection from 127.0.0.1:47656 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 15:18:49.667171 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 366
I20260927 15:18:49.668448 428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:49.791715 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 366
I20260927 15:18:49.809381 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 440
I20260927 15:18:49.810853 502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:49.936558 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 440
2026-09-27T15:18:49Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5026 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:18:50 dist-test-slave-f5h6 krb5kdc[660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:18:50 dist-test-slave-f5h6 krb5kdc[660](info): set up 2 sockets
Sep 27 15:18:50 dist-test-slave-f5h6 krb5kdc[660](info): commencing operation
krb5kdc: starting...
W20260927 15:18:50.986454 10373 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/test-user.keytab.
Sep 27 15:18:51 dist-test-slave-f5h6 krb5kdc[660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:18:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.33.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
Entry for principal kudu/127.10.33.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab.
I20260927 15:18:51.265686 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:44327
--webserver_interface=127.10.33.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:44327,127.10.33.125:43541,127.10.33.124:41865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:51.622669 673 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:18:51.623023 673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:51.623090 673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:51.623172 673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:51.630600 673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:51.630722 673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:51.630782 673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:51.630826 673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:51.630889 673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:51.641283 673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:44327,127.10.33.125:43541,127.10.33.124:41865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.126.keytab
--principal=kudu/127.10.33.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.126:44327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.33.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:51.642621 673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:51.643971 673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:51.656428 678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:51.656428 679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:51.656598 681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:51.657712 673 server_base.cc:1034] running on GCE node
I20260927 15:18:51.658598 673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:51.660133 673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:51.661391 673 hybrid_clock.cc:648] HybridClock initialized: now 1790522331661345 us; error 45 us; skew 500 ppm
I20260927 15:18:51.661826 673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:51 dist-test-slave-f5h6 krb5kdc[660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522331, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:51.665367 673 init.cc:377] Logged in from keytab as kudu/127.10.33.126@KRBTEST.COM (short username kudu)
I20260927 15:18:51.667088 673 webserver.cc:460] Webserver started at http://127.10.33.126:45959/ using document root <none> and password file <none>
I20260927 15:18:51.667805 673 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:51.667896 673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:51.668216 673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:51.671672 673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-0/data/instance:
uuid: "89bbc505e72446ec8f1b7f831733b1a1"
format_stamp: "Formatted at 2026-09-27 15:18:51 on dist-test-slave-f5h6"
server_key: "ca8e87102564c7a9583208e249ccaec1"
server_key_iv: "0aeb97a268362c6978aac3bfcf21de85"
server_key_version: "encryptionkey@0"
I20260927 15:18:51.672544 673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-0/wal/instance:
uuid: "89bbc505e72446ec8f1b7f831733b1a1"
format_stamp: "Formatted at 2026-09-27 15:18:51 on dist-test-slave-f5h6"
server_key: "ca8e87102564c7a9583208e249ccaec1"
server_key_iv: "0aeb97a268362c6978aac3bfcf21de85"
server_key_version: "encryptionkey@0"
I20260927 15:18:51.678140 673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260927 15:18:51.681958 688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:51.683670 673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:18:51.683866 673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-0/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-0/wal
uuid: "89bbc505e72446ec8f1b7f831733b1a1"
format_stamp: "Formatted at 2026-09-27 15:18:51 on dist-test-slave-f5h6"
server_key: "ca8e87102564c7a9583208e249ccaec1"
server_key_iv: "0aeb97a268362c6978aac3bfcf21de85"
server_key_version: "encryptionkey@0"
I20260927 15:18:51.684082 673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:51.706609 673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:51.711673 673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:51.711966 673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:51.733565 673 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.126:44327
I20260927 15:18:51.733543 739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.126:44327 every 8 connection(s)
I20260927 15:18:51.735026 673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:51.738371 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 673
I20260927 15:18:51.738566 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-0/wal/instance
I20260927 15:18:51.738991 10373 external_mini_cluster.cc:1428] Setting key e0a4ad3a0f4eed83721822c863e684eb
I20260927 15:18:51.739285 740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:51.746084 740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44327 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:18:51 dist-test-slave-f5h6 krb5kdc[660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522331, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.33.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.125@KRBTEST.COM" created.
I20260927 15:18:51.770648 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:51.750714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42522 (local address 127.10.33.126:44327)
0927 15:18:51.751461 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:51.751471 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:51.751533 (+ 62us) server_negotiation.cc:407] Connection header received
0927 15:18:51.752843 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:51.752850 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:51.753132 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:51.753332 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:51.754682 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:51.755900 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:51.757289 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:51.757845 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:51.758064 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:51.762277 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:51.762287 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:51.762302 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:51.762340 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:51.765933 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:51.766934 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:51.766942 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:51.766944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:51.767120 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:51.767579 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:51.767583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:51.767585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:51.768157 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:51.768381 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:51.769067 (+ 686us) server_negotiation.cc:299] Negotiation successful
0927 15:18:51.769592 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":219,"server-negotiator.queue_time_us":480,"thread_start_us":297,"threads_started":1}
I20260927 15:18:51.770696 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:51.749399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:44327 (local address 127.0.0.1:42522)
0927 15:18:51.750219 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:51.750266 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:51.752427 (+ 2161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:51.753370 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:51.753377 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:51.754120 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:51.754463 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:51.754484 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:51.756057 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:51.756061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:51.757144 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:51.757157 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:51.758116 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:51.758123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:51.758312 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:51.759038 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:51.759064 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:51.761907 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:51.766126 (+ 4219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:51.766142 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:51.766157 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:51.766740 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:51.767243 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:51.767248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:51.767251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:51.767452 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:51.768344 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:51.768352 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:51.768590 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 15:18:51.768851 (+ 261us) client_negotiation.cc:240] Negotiation successful
0927 15:18:51.769413 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":350,"threads_started":1}
I20260927 15:18:51.774107 740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:18:51.775372 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:51.774881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:43541 (local address 127.0.0.1:60968)
0927 15:18:51.775082 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:51.775220 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 15:18:51.775677 689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.33.125:43541: connect: Connection refused (error 111)
I20260927 15:18:51.777400 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:51.776971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:43541 (local address 127.0.0.1:60972)
0927 15:18:51.777148 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:51.777238 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 15:18:51.777618 740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.125:43541: Network error: Client connection negotiation failed: client connection to 127.10.33.125:43541: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.125.keytab.
Entry for principal kudu/127.10.33.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.125.keytab.
I20260927 15:18:51.798105 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.125:43541
--webserver_interface=127.10.33.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.125.keytab
--principal=kudu/127.10.33.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:44327,127.10.33.125:43541,127.10.33.124:41865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:18:51.826995 740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } attempt: 1
I20260927 15:18:51.828325 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:51.827819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:43541 (local address 127.0.0.1:60980)
0927 15:18:51.828037 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:51.828156 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 15:18:51.829330 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:51.828963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:43541 (local address 127.0.0.1:60992)
0927 15:18:51.829106 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:51.829201 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 15:18:51.829527 740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.125:43541: Network error: Client connection negotiation failed: client connection to 127.10.33.125:43541: connect: Connection refused (error 111)
I20260927 15:18:51.897794 740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } attempt: 2
I20260927 15:18:51.898900 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:51.898446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:43541 (local address 127.0.0.1:32776)
0927 15:18:51.898678 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:51.898764 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 15:18:51.899827 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:51.899492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:43541 (local address 127.0.0.1:32788)
0927 15:18:51.899633 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:51.899709 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 15:18:51.899992 740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.125:43541: Network error: Client connection negotiation failed: client connection to 127.10.33.125:43541: connect: Connection refused (error 111)
I20260927 15:18:51.991329 740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } attempt: 3
I20260927 15:18:51.992590 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:51.992081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:43541 (local address 127.0.0.1:32794)
0927 15:18:51.992311 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:51.992416 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 15:18:51.994410 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:51.993340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:43541 (local address 127.0.0.1:32796)
0927 15:18:51.994092 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:51.994197 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":386,"threads_started":1}
W20260927 15:18:51.994608 740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.125:43541: Network error: Client connection negotiation failed: client connection to 127.10.33.125:43541: connect: Connection refused (error 111)
W20260927 15:18:52.135726 747 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:18:52.136081 747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:52.136184 747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:52.136274 747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 15:18:52.137908 740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } attempt: 4
I20260927 15:18:52.139098 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.138516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:43541 (local address 127.0.0.1:32804)
0927 15:18:52.138877 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.138967 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260927 15:18:52.140226 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.139815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:43541 (local address 127.0.0.1:32810)
0927 15:18:52.139999 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.140098 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.125:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260927 15:18:52.140496 740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.125:43541: Network error: Client connection negotiation failed: client connection to 127.10.33.125:43541: connect: Connection refused (error 111)
W20260927 15:18:52.144109 747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:52.144202 747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:52.144260 747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:52.144330 747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:52.144414 747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:52.154820 747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:44327,127.10.33.125:43541,127.10.33.124:41865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.125.keytab
--principal=kudu/127.10.33.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.125:43541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.33.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:52.156134 747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:52.157460 747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:52.169113 753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:52.169313 754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:52.169497 756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:52.170388 747 server_base.cc:1034] running on GCE node
I20260927 15:18:52.171216 747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:52.172652 747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:52.173972 747 hybrid_clock.cc:648] HybridClock initialized: now 1790522332173902 us; error 38 us; skew 500 ppm
I20260927 15:18:52.174300 747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:52 dist-test-slave-f5h6 krb5kdc[660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522332, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:52.177680 747 init.cc:377] Logged in from keytab as kudu/127.10.33.125@KRBTEST.COM (short username kudu)
I20260927 15:18:52.179301 747 webserver.cc:460] Webserver started at http://127.10.33.125:39731/ using document root <none> and password file <none>
I20260927 15:18:52.180007 747 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:52.180109 747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:52.180426 747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:52.183653 747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-1/data/instance:
uuid: "3c6ef17bbdc7460284c686d6715cb365"
format_stamp: "Formatted at 2026-09-27 15:18:52 on dist-test-slave-f5h6"
server_key: "bb8316ba7bd9454d0fb5fd2e128f279f"
server_key_iv: "fa108de30c94e93629534590e1dbbd3e"
server_key_version: "encryptionkey@0"
I20260927 15:18:52.184468 747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-1/wal/instance:
uuid: "3c6ef17bbdc7460284c686d6715cb365"
format_stamp: "Formatted at 2026-09-27 15:18:52 on dist-test-slave-f5h6"
server_key: "bb8316ba7bd9454d0fb5fd2e128f279f"
server_key_iv: "fa108de30c94e93629534590e1dbbd3e"
server_key_version: "encryptionkey@0"
I20260927 15:18:52.190022 747 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 15:18:52.193887 763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:52.195350 747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:18:52.195537 747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-1/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-1/wal
uuid: "3c6ef17bbdc7460284c686d6715cb365"
format_stamp: "Formatted at 2026-09-27 15:18:52 on dist-test-slave-f5h6"
server_key: "bb8316ba7bd9454d0fb5fd2e128f279f"
server_key_iv: "fa108de30c94e93629534590e1dbbd3e"
server_key_version: "encryptionkey@0"
I20260927 15:18:52.195765 747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:52.210911 747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:52.215287 747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:52.215551 747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:52.234629 747 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.125:43541
I20260927 15:18:52.234669 814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.125:43541 every 8 connection(s)
I20260927 15:18:52.235998 747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-1/data/info.pb
I20260927 15:18:52.239794 815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:52.243140 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 747
I20260927 15:18:52.243319 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-1/wal/instance
I20260927 15:18:52.243845 10373 external_mini_cluster.cc:1428] Setting key 91a93c9051f36f67259fd70438a50db5
I20260927 15:18:52.247848 815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44327 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:18:52 dist-test-slave-f5h6 krb5kdc[660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522332, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.125@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.33.124@KRBTEST.COM; defaulting to no policy
I20260927 15:18:52.270641 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.251122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42536 (local address 127.10.33.126:44327)
0927 15:18:52.251333 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:52.251339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:52.252495 (+ 1156us) server_negotiation.cc:407] Connection header received
0927 15:18:52.254483 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:52.254489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:52.254561 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:52.254640 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:52.256716 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.257454 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.259550 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.260018 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.260125 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:52.264629 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:52.264636 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:52.264639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:52.264680 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:52.267450 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:52.268269 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:52.268274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:52.268276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:52.268401 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:52.268821 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:52.268825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:52.268827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:52.269117 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:52.269266 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:52.270065 (+ 799us) server_negotiation.cc:299] Negotiation successful
0927 15:18:52.270313 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 15:18:52.271642 817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.251516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:44327 (local address 127.0.0.1:42536)
0927 15:18:52.252322 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.252373 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:52.254167 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:52.254903 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:52.254915 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:52.255635 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:52.256450 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:52.256478 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.257618 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.257622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:52.259379 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:52.259393 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.260251 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.260257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:52.260541 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:52.261304 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:52.261338 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:52.264346 (+ 3008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:52.267613 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:52.267623 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:52.267642 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:52.268130 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:52.268511 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:52.268516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:52.268520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:52.268703 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:52.269255 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:52.269264 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:52.269671 (+ 407us) client_negotiation.cc:769] Sending connection context
0927 15:18:52.269955 (+ 284us) client_negotiation.cc:240] Negotiation successful
0927 15:18:52.270494 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":355,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.33.124@KRBTEST.COM" created.
I20260927 15:18:52.273501 815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:18:52 dist-test-slave-f5h6 krb5kdc[660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522332, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.125@KRBTEST.COM for kudu/127.10.33.125@KRBTEST.COM
I20260927 15:18:52.293304 817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.274300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:43541 (local address 127.0.0.1:32826)
0927 15:18:52.274459 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.274477 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:52.274581 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:52.278883 (+ 4302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:52.278887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:52.279344 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:52.279484 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:52.279493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.280844 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.280849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:52.281665 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:52.281676 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.282844 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.282851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:52.282990 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:52.283641 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:52.283664 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:52.286370 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:52.290574 (+ 4204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:52.290579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:52.290582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:52.291075 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:52.291559 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:52.291563 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:52.291566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:52.291727 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:52.292472 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:52.292475 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:52.292594 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 15:18:52.292712 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 15:18:52.292990 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 15:18:52.293563 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.274565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32826 (local address 127.10.33.125:43541)
0927 15:18:52.278413 (+ 3848us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:52.278420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:52.278449 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:18:52.278500 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:52.278505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:52.278709 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:52.278870 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:52.279650 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.280697 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.281797 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.282638 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.282812 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:52.286666 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:52.286686 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:52.286706 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:52.286747 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:52.290366 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:52.291228 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:52.291238 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:52.291242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:52.291406 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:52.291849 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:52.291853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:52.291856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:52.292327 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:52.292514 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:52.292881 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 15:18:52.293134 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3671,"thread_start_us":3519,"threads_started":1}
I20260927 15:18:52.296229 815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:18:52.297591 817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.297003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41865 (local address 127.0.0.1:59334)
0927 15:18:52.297236 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.297390 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 15:18:52.297955 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
I20260927 15:18:52.299543 817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.299142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41865 (local address 127.0.0.1:59348)
0927 15:18:52.299317 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.299412 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 15:18:52.299831 815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41865: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.33.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.124.keytab.
Entry for principal kudu/127.10.33.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.124.keytab.
I20260927 15:18:52.309902 10373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQegxT3/build/asan/bin/kudu
/tmp/dist-test-taskQegxT3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.124:41865
--webserver_interface=127.10.33.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.124.keytab
--principal=kudu/127.10.33.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.33.84:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.33.126:44327,127.10.33.125:43541,127.10.33.124:41865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:18:52.349164 815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } attempt: 1
I20260927 15:18:52.350461 817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.349906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41865 (local address 127.0.0.1:59358)
0927 15:18:52.350140 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.350272 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 15:18:52.351541 817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.351133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41865 (local address 127.0.0.1:59360)
0927 15:18:52.351299 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.351395 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 15:18:52.351802 815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41865: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
I20260927 15:18:52.420045 815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } attempt: 2
I20260927 15:18:52.421414 817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.420867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41865 (local address 127.0.0.1:59372)
0927 15:18:52.421092 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.421205 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 15:18:52.422533 817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.422147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41865 (local address 127.0.0.1:59382)
0927 15:18:52.422295 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.422386 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 15:18:52.422734 815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41865: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
I20260927 15:18:52.439738 740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } attempt: 5
Sep 27 15:18:52 dist-test-slave-f5h6 krb5kdc[660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522331, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for kudu/127.10.33.125@KRBTEST.COM
I20260927 15:18:52.454849 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.440752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32836 (local address 127.10.33.125:43541)
0927 15:18:52.440948 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:52.440952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:52.440971 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:18:52.441089 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:52.441093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:52.441172 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:52.441257 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:52.442360 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.443069 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.444787 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.445249 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.445362 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:52.449292 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:52.449299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:52.449302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:52.449372 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:52.451934 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:52.452793 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:52.452798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:52.452801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:52.452941 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:52.453424 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:52.453428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:52.453430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:52.453739 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:52.453907 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:52.454384 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 15:18:52.454601 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 15:18:52.454860 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.440512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:43541 (local address 127.0.0.1:32836)
0927 15:18:52.440755 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.440806 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:52.440966 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:52.441325 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:52.441331 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:52.441925 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:52.442178 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:52.442193 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.443226 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.443232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:52.444600 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:52.444618 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.445412 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.445418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:52.445655 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:52.446390 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:52.446415 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:52.449036 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:52.452084 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:52.452090 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:52.452094 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:52.452633 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:52.453066 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:52.453071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:52.453075 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:52.453278 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:52.453887 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:52.453893 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:52.454068 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 15:18:52.454243 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 15:18:52.454535 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 15:18:52.456262 740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:18:52.457394 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.456929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41865 (local address 127.0.0.1:59392)
0927 15:18:52.457123 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.457228 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 15:18:52.458518 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.458111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41865 (local address 127.0.0.1:59404)
0927 15:18:52.458309 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.458395 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 15:18:52.458745 740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41865: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
I20260927 15:18:52.510018 740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } attempt: 1
I20260927 15:18:52.511221 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.510758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41865 (local address 127.0.0.1:59420)
0927 15:18:52.510967 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.511053 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 15:18:52.512365 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.511968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41865 (local address 127.0.0.1:59430)
0927 15:18:52.512141 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.512221 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260927 15:18:52.512596 740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41865: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
I20260927 15:18:52.513929 815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } attempt: 3
I20260927 15:18:52.514923 817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.514485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41865 (local address 127.0.0.1:59436)
0927 15:18:52.514648 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.514748 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 15:18:52.515933 817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.515531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41865 (local address 127.0.0.1:59444)
0927 15:18:52.515735 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.515808 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260927 15:18:52.516126 815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41865: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
I20260927 15:18:52.580865 740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } attempt: 2
I20260927 15:18:52.582185 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.581580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41865 (local address 127.0.0.1:59452)
0927 15:18:52.581848 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.582012 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260927 15:18:52.583426 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.582999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41865 (local address 127.0.0.1:59466)
0927 15:18:52.583207 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.583301 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260927 15:18:52.583647 740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41865: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
I20260927 15:18:52.659336 815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } attempt: 4
I20260927 15:18:52.660674 817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.660079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41865 (local address 127.0.0.1:59468)
0927 15:18:52.660333 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.660426 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 15:18:52.661772 817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.661388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41865 (local address 127.0.0.1:59472)
0927 15:18:52.661547 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.661618 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 15:18:52.662010 815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41865: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
W20260927 15:18:52.664969 822 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:18:52.665323 822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:52.665416 822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:52.665526 822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:52.673125 822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:52.673273 822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:52.673368 822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:52.673431 822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:52.673516 822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:52.684018 822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.33.84:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.33.126:44327,127.10.33.125:43541,127.10.33.124:41865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/krb5kdc/kudu_127.10.33.124.keytab
--principal=kudu/127.10.33.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.33.124:41865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.33.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 13:46:26 UTC on ccycloud.quasar-xkqhuq.root.comops.site
build id 4676080
ASAN enabled
I20260927 15:18:52.685425 822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:52.687031 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:18:52.689899 740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } attempt: 3
I20260927 15:18:52.691136 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.690540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41865 (local address 127.0.0.1:59476)
0927 15:18:52.690874 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.690970 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260927 15:18:52.692245 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.691869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41865 (local address 127.0.0.1:59482)
0927 15:18:52.692041 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.692124 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 15:18:52.692485 740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.33.124:41865: Network error: Client connection negotiation failed: client connection to 127.10.33.124:41865: connect: Connection refused (error 111)
W20260927 15:18:52.699553 827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:52.700026 828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:52.700047 830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:52.701372 822 server_base.cc:1034] running on GCE node
I20260927 15:18:52.702440 822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:52.704017 822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:52.705297 822 hybrid_clock.cc:648] HybridClock initialized: now 1790522332705174 us; error 66 us; skew 500 ppm
I20260927 15:18:52.705817 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 15:18:52 dist-test-slave-f5h6 krb5kdc[660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522332, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:52.709244 822 init.cc:377] Logged in from keytab as kudu/127.10.33.124@KRBTEST.COM (short username kudu)
I20260927 15:18:52.710929 822 webserver.cc:460] Webserver started at http://127.10.33.124:35863/ using document root <none> and password file <none>
I20260927 15:18:52.711611 822 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:52.711709 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:52.711987 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:18:52.715160 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-2/data/instance:
uuid: "f7cd691e98c64c86a7dd5d952d36be17"
format_stamp: "Formatted at 2026-09-27 15:18:52 on dist-test-slave-f5h6"
server_key: "30df16c59bb937b0e5752ad126bdec30"
server_key_iv: "e9040b9ef835befd0387c1065b4fea53"
server_key_version: "encryptionkey@0"
I20260927 15:18:52.715992 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-2/wal/instance:
uuid: "f7cd691e98c64c86a7dd5d952d36be17"
format_stamp: "Formatted at 2026-09-27 15:18:52 on dist-test-slave-f5h6"
server_key: "30df16c59bb937b0e5752ad126bdec30"
server_key_iv: "e9040b9ef835befd0387c1065b4fea53"
server_key_version: "encryptionkey@0"
I20260927 15:18:52.721297 822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:18:52.725469 837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:52.727100 822 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:18:52.727275 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-2/data,/tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-2/wal
uuid: "f7cd691e98c64c86a7dd5d952d36be17"
format_stamp: "Formatted at 2026-09-27 15:18:52 on dist-test-slave-f5h6"
server_key: "30df16c59bb937b0e5752ad126bdec30"
server_key_iv: "e9040b9ef835befd0387c1065b4fea53"
server_key_version: "encryptionkey@0"
I20260927 15:18:52.727578 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:52.746512 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:52.751008 822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:52.751242 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:52.768882 822 rpc_server.cc:307] RPC server started. Bound to: 127.10.33.124:41865
I20260927 15:18:52.768911 888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.33.124:41865 every 8 connection(s)
I20260927 15:18:52.770277 822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-2/data/info.pb
I20260927 15:18:52.774081 889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:52.776345 10373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQegxT3/build/asan/bin/kudu as pid 822
I20260927 15:18:52.776504 10373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQegxT3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521968392480-10373-0/minicluster-data/master-2/wal/instance
I20260927 15:18:52.776919 10373 external_mini_cluster.cc:1428] Setting key 1af53cefb1931d9acf5f00fb0c97c61a
I20260927 15:18:52.781739 889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44327 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:18:52 dist-test-slave-f5h6 krb5kdc[660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.124@KRBTEST.COM
Sep 27 15:18:52 dist-test-slave-f5h6 krb5kdc[660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522332, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.124@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:18:52.800450 893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.784929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42552 (local address 127.10.33.126:44327)
0927 15:18:52.785489 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:52.785496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:52.786018 (+ 522us) server_negotiation.cc:407] Connection header received
0927 15:18:52.786222 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:52.786227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:52.786309 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:52.786417 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:52.787553 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.788401 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.789728 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.790184 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.790295 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:52.793979 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:52.793985 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:52.793988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:52.794025 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:52.797119 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:52.797956 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:52.797962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:52.797965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:52.798130 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:52.798610 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:52.798614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:52.798617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:52.799051 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:52.799195 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:52.799779 (+ 584us) server_negotiation.cc:299] Negotiation successful
0927 15:18:52.800046 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":281,"threads_started":1}
I20260927 15:18:52.800834 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.778647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59484 (local address 127.10.33.124:41865)
0927 15:18:52.779465 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:52.779477 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:52.779529 (+ 52us) server_negotiation.cc:407] Connection header received
0927 15:18:52.781262 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:52.781287 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:52.781733 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:52.782154 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:52.784043 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.785513 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.787029 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.787774 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.787948 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:52.791868 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:52.791880 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:52.791899 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:52.791937 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:52.795896 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:52.796732 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:52.796741 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:52.796752 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:52.796975 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:52.797443 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:52.797447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:52.797449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:52.798180 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:52.798442 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:52.799030 (+ 588us) server_negotiation.cc:299] Negotiation successful
0927 15:18:52.799523 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":352,"threads_started":1}
I20260927 15:18:52.800884 894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.784947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.126:44327 (local address 127.0.0.1:42552)
0927 15:18:52.785879 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.785929 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:52.786085 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:52.786475 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:52.786480 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:52.787110 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:52.787404 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:52.787413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.788544 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.788548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:52.789588 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:52.789600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.790403 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.790410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:52.790548 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:52.791119 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:52.791143 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:52.793591 (+ 2448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:52.797301 (+ 3710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:52.797317 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:52.797325 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:52.797773 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:52.798279 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:52.798284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:52.798288 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:52.798467 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:52.799251 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:52.799260 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:52.799457 (+ 197us) client_negotiation.cc:769] Sending connection context
0927 15:18:52.799625 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 15:18:52.799868 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":521,"threads_started":1}
I20260927 15:18:52.802582 889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:18:52 dist-test-slave-f5h6 krb5kdc[660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522332, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.124@KRBTEST.COM for kudu/127.10.33.125@KRBTEST.COM
Sep 27 15:18:52 dist-test-slave-f5h6 krb5kdc[660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.126@KRBTEST.COM
I20260927 15:18:52.817402 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.803450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32838 (local address 127.10.33.125:43541)
0927 15:18:52.803631 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:52.803636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:52.804219 (+ 583us) server_negotiation.cc:407] Connection header received
0927 15:18:52.804485 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:52.804490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:52.804566 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:52.804670 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:52.805336 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.806264 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.807740 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.808556 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.808690 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:52.811849 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:52.811854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:52.811857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:52.811890 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:52.814854 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:52.815544 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:52.815550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:52.815553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:52.815708 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:52.816111 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:52.816116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:52.816119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:52.816461 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:52.816622 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:52.816961 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 15:18:52.817127 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:18:52.817493 894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.803453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.125:43541 (local address 127.0.0.1:32838)
0927 15:18:52.804134 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.804152 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:52.804309 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:52.804702 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:52.804706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:52.805051 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:52.805195 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:52.805203 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.806399 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.806405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:52.807573 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:52.807585 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.808756 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.808762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:52.808971 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:52.809603 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:52.809625 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:52.811665 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:52.814997 (+ 3332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:52.815003 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:52.815007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:52.815429 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:52.815829 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:52.815834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:52.815838 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:52.816003 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:52.816591 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:52.816594 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:52.816732 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 15:18:52.816881 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 15:18:52.817160 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574}
I20260927 15:18:52.818495 889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:18:52.819501 893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.804004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42564 (local address 127.10.33.126:44327)
0927 15:18:52.804174 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:52.804179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:52.804197 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:18:52.804385 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:52.804391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:52.804483 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:52.804554 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:52.805994 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.806956 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.808714 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.809401 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.809531 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:52.813323 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:52.813328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:52.813331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:52.813373 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:52.816692 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:52.817486 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:52.817492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:52.817496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:52.817646 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:52.818089 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:52.818094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:52.818098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:52.818379 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:52.818638 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:52.818982 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 15:18:52.819212 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 15:18:52 dist-test-slave-f5h6 krb5kdc[660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522332, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.124@KRBTEST.COM for kudu/127.10.33.124@KRBTEST.COM
Sep 27 15:18:52 dist-test-slave-f5h6 krb5kdc[660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.33.125@KRBTEST.COM
I20260927 15:18:52.832332 894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.819126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41865 (local address 127.0.0.1:59490)
0927 15:18:52.819364 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.819380 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:52.819462 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:52.819815 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:52.819818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:52.820269 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:52.820439 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:52.820448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.821628 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.821634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:52.822624 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:52.822636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.823401 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.823407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:52.823615 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:52.824218 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:52.824260 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:52.826396 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:52.829632 (+ 3236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:52.829636 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:52.829639 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:52.830054 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:52.830436 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:52.830441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:52.830444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:52.830615 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:52.831247 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:52.831252 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:52.831405 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 15:18:52.831583 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 15:18:52.831917 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260927 15:18:52.832393 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.819289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59490 (local address 127.10.33.124:41865)
0927 15:18:52.819527 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:52.819533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:52.819549 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:18:52.819607 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:52.819613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:52.819694 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:52.819789 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:52.820559 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.821510 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.822760 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.823258 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.823351 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:52.826587 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:52.826594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:52.826599 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:52.826636 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:52.829504 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:52.830165 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:52.830171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:52.830175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:52.830339 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:52.830775 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:52.830779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:52.830781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:52.831079 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:52.831217 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:52.831735 (+ 518us) server_negotiation.cc:299] Negotiation successful
0927 15:18:52.832075 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":125}
I20260927 15:18:52.836139 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.821280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32848 (local address 127.10.33.125:43541)
0927 15:18:52.821424 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:52.821428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:52.821444 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:18:52.821583 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:52.821588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:52.821662 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:52.821785 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:52.822938 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.823830 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.825070 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.825526 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.825620 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:52.830565 (+ 4945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:52.830571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:52.830576 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:52.830617 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:52.833495 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:52.834230 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:52.834236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:52.834239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:52.834376 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:52.834741 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:52.834745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:52.834747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:52.835018 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:52.835279 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:52.835585 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 15:18:52.835836 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:18:52.837572 889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7cd691e98c64c86a7dd5d952d36be17: Bootstrap starting.
I20260927 15:18:52.841099 10373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:18:52.841203 889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7cd691e98c64c86a7dd5d952d36be17: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:52.842355 889 log.cc:824] T 00000000000000000000000000000000 P f7cd691e98c64c86a7dd5d952d36be17: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:52.845005 889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7cd691e98c64c86a7dd5d952d36be17: No bootstrap required, opened a new log
I20260927 15:18:52.843451 715 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "89bbc505e72446ec8f1b7f831733b1a1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42564
I20260927 15:18:52.852578 889 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7cd691e98c64c86a7dd5d952d36be17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bbc505e72446ec8f1b7f831733b1a1" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44327 } } peers { permanent_uuid: "3c6ef17bbdc7460284c686d6715cb365" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } } peers { permanent_uuid: "f7cd691e98c64c86a7dd5d952d36be17" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } }
I20260927 15:18:52.852946 889 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7cd691e98c64c86a7dd5d952d36be17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7cd691e98c64c86a7dd5d952d36be17, State: Initialized, Role: FOLLOWER
I20260927 15:18:52.853394 889 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7cd691e98c64c86a7dd5d952d36be17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bbc505e72446ec8f1b7f831733b1a1" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44327 } } peers { permanent_uuid: "3c6ef17bbdc7460284c686d6715cb365" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } } peers { permanent_uuid: "f7cd691e98c64c86a7dd5d952d36be17" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } }
I20260927 15:18:52.854300 896 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7cd691e98c64c86a7dd5d952d36be17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bbc505e72446ec8f1b7f831733b1a1" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44327 } } peers { permanent_uuid: "3c6ef17bbdc7460284c686d6715cb365" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } } peers { permanent_uuid: "f7cd691e98c64c86a7dd5d952d36be17" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } } }
I20260927 15:18:52.854606 896 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7cd691e98c64c86a7dd5d952d36be17 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:18:52.855624 889 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7cd691e98c64c86a7dd5d952d36be17 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 15:18:52.865000 907 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7cd691e98c64c86a7dd5d952d36be17: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:18:52.865159 907 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:18:52.869650 740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } attempt: 4
I20260927 15:18:52.862813 822 master_runner.cc:186] Error getting master registration for 127.10.33.126:44327: OK, instance_id { permanent_uuid: "89bbc505e72446ec8f1b7f831733b1a1" instance_seqno: 1790522331725168 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:18:52.874135 822 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 15:18:52 dist-test-slave-f5h6 krb5kdc[660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522331, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.126@KRBTEST.COM for kudu/127.10.33.124@KRBTEST.COM
I20260927 15:18:52.886518 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.870456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41865 (local address 127.0.0.1:59502)
0927 15:18:52.870670 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.870693 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:52.870822 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:52.871316 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:52.871322 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:52.871832 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:52.872051 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:52.872064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.873700 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.873706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:52.874829 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:52.874854 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.875877 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.875883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:52.876076 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:52.876754 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:52.876776 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:52.879422 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:52.883505 (+ 4083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:52.883511 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:52.883515 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:52.884038 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:52.884450 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:52.884455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:52.884459 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:52.884638 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:52.885322 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:52.885327 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:52.885518 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 15:18:52.885660 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 15:18:52.886117 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 15:18:52.886565 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.870777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59502 (local address 127.10.33.124:41865)
0927 15:18:52.870989 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:52.870996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:52.871016 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:18:52.871082 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:52.871087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:52.871166 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:52.871280 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:52.872231 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.873532 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.874986 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.875730 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.875867 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:52.879646 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:52.879653 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:52.879658 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:52.879702 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:52.883353 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:52.884171 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:52.884177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:52.884181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:52.884343 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:52.884745 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:52.884750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:52.884754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:52.885203 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:52.885383 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:52.885799 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 15:18:52.886184 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:18:52.891397 740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1: Bootstrap starting.
I20260927 15:18:52.895013 740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:52.896118 740 log.cc:824] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:52.898950 740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1: No bootstrap required, opened a new log
I20260927 15:18:52.904547 740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bbc505e72446ec8f1b7f831733b1a1" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44327 } } peers { permanent_uuid: "3c6ef17bbdc7460284c686d6715cb365" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } } peers { permanent_uuid: "f7cd691e98c64c86a7dd5d952d36be17" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } }
I20260927 15:18:52.904935 740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89bbc505e72446ec8f1b7f831733b1a1, State: Initialized, Role: FOLLOWER
I20260927 15:18:52.905395 740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bbc505e72446ec8f1b7f831733b1a1" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44327 } } peers { permanent_uuid: "3c6ef17bbdc7460284c686d6715cb365" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } } peers { permanent_uuid: "f7cd691e98c64c86a7dd5d952d36be17" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } }
I20260927 15:18:52.906584 910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bbc505e72446ec8f1b7f831733b1a1" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44327 } } peers { permanent_uuid: "3c6ef17bbdc7460284c686d6715cb365" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } } peers { permanent_uuid: "f7cd691e98c64c86a7dd5d952d36be17" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } } }
I20260927 15:18:52.907092 910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:18:52.908057 740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:52.908200 715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 15:18:52.908416 715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:52.912330 715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bbc505e72446ec8f1b7f831733b1a1" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44327 } } peers { permanent_uuid: "3c6ef17bbdc7460284c686d6715cb365" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } } peers { permanent_uuid: "f7cd691e98c64c86a7dd5d952d36be17" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } }
I20260927 15:18:52.916038 864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89bbc505e72446ec8f1b7f831733b1a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f7cd691e98c64c86a7dd5d952d36be17"
I20260927 15:18:52.916616 864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7cd691e98c64c86a7dd5d952d36be17 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:52.920295 864 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7cd691e98c64c86a7dd5d952d36be17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89bbc505e72446ec8f1b7f831733b1a1 in term 1.
W20260927 15:18:52.923815 921 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:18:52.923986 921 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:18:52.923918 689 leader_election.cc:302] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89bbc505e72446ec8f1b7f831733b1a1, f7cd691e98c64c86a7dd5d952d36be17; no voters:
I20260927 15:18:52.924714 910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:52.924912 715 leader_election.cc:288] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1 [CANDIDATE]: Term 1 election: Requested vote from peers 3c6ef17bbdc7460284c686d6715cb365 (127.10.33.125:43541), f7cd691e98c64c86a7dd5d952d36be17 (127.10.33.124:41865)
I20260927 15:18:52.928989 673 master_runner.cc:186] Error getting master registration for 127.10.33.125:43541: OK, instance_id { permanent_uuid: "3c6ef17bbdc7460284c686d6715cb365" instance_seqno: 1790522332229392 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:18:52.948496 673 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:18:52.954108 910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1 [term 1 LEADER]: Becoming Leader. State: Replica: 89bbc505e72446ec8f1b7f831733b1a1, State: Running, Role: LEADER
I20260927 15:18:52.955276 910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bbc505e72446ec8f1b7f831733b1a1" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44327 } } peers { permanent_uuid: "3c6ef17bbdc7460284c686d6715cb365" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } } peers { permanent_uuid: "f7cd691e98c64c86a7dd5d952d36be17" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } }
I20260927 15:18:52.961215 815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } attempt: 5
W20260927 15:18:52.966621 694 tablet.cc:2369] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 15:18:52 dist-test-slave-f5h6 krb5kdc[660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522332, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.33.125@KRBTEST.COM for kudu/127.10.33.124@KRBTEST.COM
I20260927 15:18:52.986512 817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.962000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.33.124:41865 (local address 127.0.0.1:59506)
0927 15:18:52.962477 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:52.962500 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:52.962640 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:52.963568 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:52.963574 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:52.964157 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:52.964406 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:52.964419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.968329 (+ 3910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.968336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:52.969804 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:52.969827 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.971072 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.971080 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:52.971351 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:52.975727 (+ 4376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:52.975759 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:52.978911 (+ 3152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:52.983397 (+ 4486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:52.983404 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:52.983408 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:52.984022 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:52.984513 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:52.984519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:52.984523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:52.984687 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:52.985399 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:52.985405 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:52.985570 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 15:18:52.985705 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 15:18:52.986065 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326}
I20260927 15:18:52.987533 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.962871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59506 (local address 127.10.33.124:41865)
0927 15:18:52.963058 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:52.963064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:52.963084 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:18:52.963157 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:52.963163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:52.963257 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:52.963354 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:52.964650 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.968008 (+ 3358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.970042 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:52.970698 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:52.970831 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:52.979249 (+ 8418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:52.979257 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:52.979262 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:52.979314 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:52.983128 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:52.984189 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:52.984195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:52.984200 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:52.984380 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:52.984806 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:52.984810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:52.984814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:52.985177 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:52.985318 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:52.986942 (+ 1624us) server_negotiation.cc:299] Negotiation successful
0927 15:18:52.987209 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:18:52.994946 815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c6ef17bbdc7460284c686d6715cb365: Bootstrap starting.
W20260927 15:18:53.003490 843 tablet.cc:2369] T 00000000000000000000000000000000 P f7cd691e98c64c86a7dd5d952d36be17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:18:53.009892 815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c6ef17bbdc7460284c686d6715cb365: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:53.011626 815 log.cc:824] T 00000000000000000000000000000000 P 3c6ef17bbdc7460284c686d6715cb365: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:53.013954 923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89bbc505e72446ec8f1b7f831733b1a1. Latest consensus state: current_term: 1 leader_uuid: "89bbc505e72446ec8f1b7f831733b1a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bbc505e72446ec8f1b7f831733b1a1" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44327 } } peers { permanent_uuid: "3c6ef17bbdc7460284c686d6715cb365" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } } peers { permanent_uuid: "f7cd691e98c64c86a7dd5d952d36be17" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } } }
I20260927 15:18:53.014622 923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:53.020627 815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c6ef17bbdc7460284c686d6715cb365: No bootstrap required, opened a new log
I20260927 15:18:53.029742 934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:53.032390 934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:53.033334 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.988029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.10.33.124:41865)
0927 15:18:52.994199 (+ 6170us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:52.994207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.013520 (+ 19313us) server_negotiation.cc:407] Connection header received
0927 15:18:53.014095 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.014102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.014190 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.014264 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:53.016524 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.017550 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.019440 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.020124 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.020231 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:53.023721 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:53.023728 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:53.023732 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:53.023779 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:53.027527 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:53.029146 (+ 1619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:53.029152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:53.029155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:53.029323 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:53.030128 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:53.030134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:53.030136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:53.030575 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:53.030982 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:53.032056 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0927 15:18:53.032613 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6033,"thread_start_us":13547,"threads_started":1}
I20260927 15:18:53.042354 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.972251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.10.33.125:43541)
0927 15:18:52.972424 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:52.972430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.000915 (+ 28485us) server_negotiation.cc:407] Connection header received
0927 15:18:53.004796 (+ 3881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.004803 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.004904 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.004979 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:53.007058 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.008189 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.015096 (+ 6907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.015840 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.015955 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:53.032811 (+ 16856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:53.032820 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:53.032825 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:53.032875 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:53.036834 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:53.038193 (+ 1359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:53.038221 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:53.038225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:53.038505 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:53.039273 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:53.039279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:53.039304 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:53.039922 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:53.040332 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:53.041323 (+ 991us) server_negotiation.cc:299] Negotiation successful
0927 15:18:53.041773 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:18:53.044337 815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c6ef17bbdc7460284c686d6715cb365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bbc505e72446ec8f1b7f831733b1a1" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44327 } } peers { permanent_uuid: "3c6ef17bbdc7460284c686d6715cb365" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } } peers { permanent_uuid: "f7cd691e98c64c86a7dd5d952d36be17" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } }
I20260927 15:18:53.044682 893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:52.970268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42574 (local address 127.10.33.126:44327)
0927 15:18:52.970475 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:52.970480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.001806 (+ 31326us) server_negotiation.cc:407] Connection header received
0927 15:18:53.002314 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.002320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.002412 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.002486 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:53.010137 (+ 7651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.011237 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.016299 (+ 5062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.017115 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.017241 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:53.024062 (+ 6821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:53.024069 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:53.024074 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:53.024120 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:53.028279 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:53.037045 (+ 8766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:53.037051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:53.037054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:53.037295 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:53.042284 (+ 4989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:53.042289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:53.042293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:53.042718 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:53.043018 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:53.044028 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0927 15:18:53.044296 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:18:53.044759 815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c6ef17bbdc7460284c686d6715cb365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c6ef17bbdc7460284c686d6715cb365, State: Initialized, Role: FOLLOWER
I20260927 15:18:53.045303 815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c6ef17bbdc7460284c686d6715cb365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bbc505e72446ec8f1b7f831733b1a1" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44327 } } peers { permanent_uuid: "3c6ef17bbdc7460284c686d6715cb365" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } } peers { permanent_uuid: "f7cd691e98c64c86a7dd5d952d36be17" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } }
I20260927 15:18:53.046494 936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c6ef17bbdc7460284c686d6715cb365 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bbc505e72446ec8f1b7f831733b1a1" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44327 } } peers { permanent_uuid: "3c6ef17bbdc7460284c686d6715cb365" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } } peers { permanent_uuid: "f7cd691e98c64c86a7dd5d952d36be17" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } } }
I20260927 15:18:53.047159 936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c6ef17bbdc7460284c686d6715cb365 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:18:53.052775 864 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7cd691e98c64c86a7dd5d952d36be17 [term 1 FOLLOWER]: Refusing update from remote peer 89bbc505e72446ec8f1b7f831733b1a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:18:53.054843 815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c6ef17bbdc7460284c686d6715cb365 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:53.055291 789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c6ef17bbdc7460284c686d6715cb365 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:53.057097 923 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7cd691e98c64c86a7dd5d952d36be17" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:18:53.057353 790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89bbc505e72446ec8f1b7f831733b1a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3c6ef17bbdc7460284c686d6715cb365"
I20260927 15:18:53.060009 789 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c6ef17bbdc7460284c686d6715cb365 [term 1 FOLLOWER]: Refusing update from remote peer 89bbc505e72446ec8f1b7f831733b1a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:18:53.061137 923 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c6ef17bbdc7460284c686d6715cb365" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:18:53.063813 790 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c6ef17bbdc7460284c686d6715cb365 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89bbc505e72446ec8f1b7f831733b1a1 in term 1.
I20260927 15:18:53.080313 896 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7cd691e98c64c86a7dd5d952d36be17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89bbc505e72446ec8f1b7f831733b1a1. Latest consensus state: current_term: 1 leader_uuid: "89bbc505e72446ec8f1b7f831733b1a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bbc505e72446ec8f1b7f831733b1a1" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44327 } } peers { permanent_uuid: "3c6ef17bbdc7460284c686d6715cb365" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } } peers { permanent_uuid: "f7cd691e98c64c86a7dd5d952d36be17" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } } }
I20260927 15:18:53.080634 896 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7cd691e98c64c86a7dd5d952d36be17 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:18:53.081849 936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c6ef17bbdc7460284c686d6715cb365 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89bbc505e72446ec8f1b7f831733b1a1. Latest consensus state: current_term: 1 leader_uuid: "89bbc505e72446ec8f1b7f831733b1a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bbc505e72446ec8f1b7f831733b1a1" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44327 } } peers { permanent_uuid: "3c6ef17bbdc7460284c686d6715cb365" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } } peers { permanent_uuid: "f7cd691e98c64c86a7dd5d952d36be17" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } } }
I20260927 15:18:53.082186 936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c6ef17bbdc7460284c686d6715cb365 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:18:53.087143 935 mvcc.cc:205] Tried to move back new op lower bound from 7333979476171558912 to 7333979475795636224. Current Snapshot: MvccSnapshot[applied={T|T < 7333979476171558912}]
I20260927 15:18:53.088032 910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "89bbc505e72446ec8f1b7f831733b1a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bbc505e72446ec8f1b7f831733b1a1" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44327 } } peers { permanent_uuid: "3c6ef17bbdc7460284c686d6715cb365" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } } peers { permanent_uuid: "f7cd691e98c64c86a7dd5d952d36be17" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } } }
I20260927 15:18:53.088392 910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:53.094995 934 catalog_manager.cc:1347] Generated new cluster ID: 0c00df4f843842478bcd84dcea9e723e
I20260927 15:18:53.095127 934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:53.101383 923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "89bbc505e72446ec8f1b7f831733b1a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bbc505e72446ec8f1b7f831733b1a1" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44327 } } peers { permanent_uuid: "3c6ef17bbdc7460284c686d6715cb365" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } } peers { permanent_uuid: "f7cd691e98c64c86a7dd5d952d36be17" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } } }
I20260927 15:18:53.101701 923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1 [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:53.112823 936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c6ef17bbdc7460284c686d6715cb365 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "89bbc505e72446ec8f1b7f831733b1a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bbc505e72446ec8f1b7f831733b1a1" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44327 } } peers { permanent_uuid: "3c6ef17bbdc7460284c686d6715cb365" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } } peers { permanent_uuid: "f7cd691e98c64c86a7dd5d952d36be17" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } } }
I20260927 15:18:53.113140 936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c6ef17bbdc7460284c686d6715cb365 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:18:53.122058 896 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7cd691e98c64c86a7dd5d952d36be17 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "89bbc505e72446ec8f1b7f831733b1a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bbc505e72446ec8f1b7f831733b1a1" member_type: VOTER last_known_addr { host: "127.10.33.126" port: 44327 } } peers { permanent_uuid: "3c6ef17bbdc7460284c686d6715cb365" member_type: VOTER last_known_addr { host: "127.10.33.125" port: 43541 } } peers { permanent_uuid: "f7cd691e98c64c86a7dd5d952d36be17" member_type: VOTER last_known_addr { host: "127.10.33.124" port: 41865 } } }
I20260927 15:18:53.122383 896 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7cd691e98c64c86a7dd5d952d36be17 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:18:53.147089 934 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:53.149055 934 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:53.171952 934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89bbc505e72446ec8f1b7f831733b1a1: Generated new TSK 0
I20260927 15:18:53.173136 934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:18:53.190824 955 catalog_manager.cc:1259] Loaded cluster ID: 0c00df4f843842478bcd84dcea9e723e
I20260927 15:18:53.190948 955 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3c6ef17bbdc7460284c686d6715cb365: loading cluster ID for follower catalog manager: success
I20260927 15:18:53.194458 955 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3c6ef17bbdc7460284c686d6715cb365: acquiring CA information for follower catalog manager: success
I20260927 15:18:53.196898 955 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3c6ef17bbdc7460284c686d6715cb365: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 15:18:53.310005 893 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:53.287622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42580 (local address 127.10.33.126:44327)
0927 15:18:53.287954 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.287961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.304025 (+ 16064us) server_negotiation.cc:407] Connection header received
0927 15:18:53.308287 (+ 4262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.308295 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.308405 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.308484 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:53.309711 (+ 1227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42580: BlockingRecv error: recv got EOF from 127.0.0.1:42580 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 15:18:53.313130 819 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:53.289264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32876 (local address 127.10.33.125:43541)
0927 15:18:53.289490 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.289496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.311099 (+ 21603us) server_negotiation.cc:407] Connection header received
0927 15:18:53.311547 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.311554 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.311663 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.311736 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:53.312887 (+ 1151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32876: BlockingRecv error: recv got EOF from 127.0.0.1:32876 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 15:18:53.315558 892 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:53.304679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.10.33.124:41865)
0927 15:18:53.304870 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.304876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.313770 (+ 8894us) server_negotiation.cc:407] Connection header received
0927 15:18:53.314040 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.314047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.314135 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.314203 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:53.315266 (+ 1063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59524: BlockingRecv error: recv got EOF from 127.0.0.1:59524 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 15:18:53.366768 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:53.347439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32882 (local address 127.10.33.125:43541)
0927 15:18:53.347634 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.347640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.358691 (+ 11051us) server_negotiation.cc:407] Connection header received
0927 15:18:53.359093 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.359100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.359332 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.359485 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:53.360330 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.361536 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.363587 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.364287 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.364393 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:53.365171 (+ 778us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:53.365947 (+ 776us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:53.366070 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:53.366308 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 15:18:53.366419 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 15:18:53.411983 892 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:53.400055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59536 (local address 127.10.33.124:41865)
0927 15:18:53.400259 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.400266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.409479 (+ 9213us) server_negotiation.cc:407] Connection header received
0927 15:18:53.410091 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.410098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.410242 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.410326 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:53.411713 (+ 1387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59536: BlockingRecv error: recv got EOF from 127.0.0.1:59536 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:18:53.465574 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:53.447484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.10.33.125:43541)
0927 15:18:53.447694 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.447701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.458460 (+ 10759us) server_negotiation.cc:407] Connection header received
0927 15:18:53.459073 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.459081 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.459203 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.459275 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:53.459976 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.461163 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.463235 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.463947 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.464048 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:53.464688 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:53.464861 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:53.464925 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:53.465150 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 15:18:53.465245 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 15:18:53.512744 892 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:53.499298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59538 (local address 127.10.33.124:41865)
0927 15:18:53.499536 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.499542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.510928 (+ 11386us) server_negotiation.cc:407] Connection header received
0927 15:18:53.511400 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.511407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.511502 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.511570 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:53.512512 (+ 942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59538: BlockingRecv error: recv got EOF from 127.0.0.1:59538 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 15:18:53.541177 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:53.533997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32894 (local address 127.10.33.125:43541)
0927 15:18:53.534197 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.534204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.534743 (+ 539us) server_negotiation.cc:407] Connection header received
0927 15:18:53.534952 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.534958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.535230 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.535387 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:53.535853 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.537117 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.539117 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.539741 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.539872 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:53.540358 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:53.540507 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:53.540594 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:53.540735 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 15:18:53.540825 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 15:18:53.551028 892 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:53.548866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59548 (local address 127.10.33.124:41865)
0927 15:18:53.549064 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.549070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.549454 (+ 384us) server_negotiation.cc:407] Connection header received
0927 15:18:53.549647 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.549651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.549727 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.549824 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:53.550772 (+ 948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59548: BlockingRecv error: recv got EOF from 127.0.0.1:59548 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 15:18:53.564962 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:53.558915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32896 (local address 127.10.33.125:43541)
0927 15:18:53.559161 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.559167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.559486 (+ 319us) server_negotiation.cc:407] Connection header received
0927 15:18:53.559625 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.559631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.559724 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.559832 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:53.560210 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.561310 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.562844 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.563533 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.563662 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:53.564106 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:53.564258 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:53.564348 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:53.564575 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 15:18:53.564667 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
W20260927 15:18:53.573342 892 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:53.571402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59550 (local address 127.10.33.124:41865)
0927 15:18:53.571664 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.571669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.571856 (+ 187us) server_negotiation.cc:407] Connection header received
0927 15:18:53.572049 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.572052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.572136 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.572217 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:53.573091 (+ 874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59550: BlockingRecv error: recv got EOF from 127.0.0.1:59550 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
I20260927 15:18:53.591753 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:53.585331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32906 (local address 127.10.33.125:43541)
0927 15:18:53.585561 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.585568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.586005 (+ 437us) server_negotiation.cc:407] Connection header received
0927 15:18:53.586256 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.586263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.586356 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.586464 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:53.586907 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.588050 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.589645 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.590356 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.590524 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:53.590950 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:53.591116 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:53.591208 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:53.591340 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 15:18:53.591424 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 15:18:53.601449 892 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:53.598017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59554 (local address 127.10.33.124:41865)
0927 15:18:53.598164 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.598168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.599848 (+ 1680us) server_negotiation.cc:407] Connection header received
0927 15:18:53.600093 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.600099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.600236 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.600347 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:53.601222 (+ 875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59554: BlockingRecv error: recv got EOF from 127.0.0.1:59554 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:18:53.631078 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:53.624806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32914 (local address 127.10.33.125:43541)
0927 15:18:53.625004 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.625010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.625327 (+ 317us) server_negotiation.cc:407] Connection header received
0927 15:18:53.625579 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.625584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.625730 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.625832 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:53.626396 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.627209 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.629065 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.629541 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.629653 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:53.630240 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:53.630360 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:53.630449 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:53.630645 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 15:18:53.630800 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260927 15:18:53.640491 892 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:53.638513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59568 (local address 127.10.33.124:41865)
0927 15:18:53.638715 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.638722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.638991 (+ 269us) server_negotiation.cc:407] Connection header received
0927 15:18:53.639147 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.639153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.639264 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.639385 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:53.640244 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59568: BlockingRecv error: recv got EOF from 127.0.0.1:59568 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 15:18:53.681389 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:53.676328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.10.33.125:43541)
0927 15:18:53.676518 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.676529 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.676828 (+ 299us) server_negotiation.cc:407] Connection header received
0927 15:18:53.677027 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.677032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.677103 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.677188 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:53.677600 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.678388 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.679705 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.680150 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.680264 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:53.680672 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:53.680784 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:53.680860 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:53.680996 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 15:18:53.681086 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 15:18:53.691685 892 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:53.689411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59582 (local address 127.10.33.124:41865)
0927 15:18:53.689645 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.689653 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.690052 (+ 399us) server_negotiation.cc:407] Connection header received
0927 15:18:53.690263 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.690268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.690374 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.690472 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:53.691446 (+ 974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59582: BlockingRecv error: recv got EOF from 127.0.0.1:59582 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:18:53.764657 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:53.759564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32940 (local address 127.10.33.125:43541)
0927 15:18:53.759776 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.759782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.760127 (+ 345us) server_negotiation.cc:407] Connection header received
0927 15:18:53.760340 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.760345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.760413 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.760489 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:53.760893 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.761613 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.762974 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.763424 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.763547 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:53.763938 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:53.764045 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:53.764120 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:53.764272 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 15:18:53.764359 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 15:18:53.783587 892 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:18:53.771727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.10.33.124:41865)
0927 15:18:53.771959 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.771967 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.782104 (+ 10137us) server_negotiation.cc:407] Connection header received
0927 15:18:53.782333 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.782338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.782437 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.782534 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:53.783350 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59584: BlockingRecv error: recv got EOF from 127.0.0.1:59584 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 15:18:53.867314 907 catalog_manager.cc:1259] Loaded cluster ID: 0c00df4f843842478bcd84dcea9e723e
I20260927 15:18:53.867434 907 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7cd691e98c64c86a7dd5d952d36be17: loading cluster ID for follower catalog manager: success
I20260927 15:18:53.870146 907 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7cd691e98c64c86a7dd5d952d36be17: acquiring CA information for follower catalog manager: success
I20260927 15:18:53.872061 907 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f7cd691e98c64c86a7dd5d952d36be17: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:18:53.921021 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:53.915548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32952 (local address 127.10.33.125:43541)
0927 15:18:53.915854 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.915862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.916021 (+ 159us) server_negotiation.cc:407] Connection header received
0927 15:18:53.916169 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.916173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.916256 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.916387 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:53.916851 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.917730 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.919202 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.919650 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.919787 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:53.920175 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:53.920368 (+ 193us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:53.920453 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:53.920555 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 15:18:53.920682 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260927 15:18:53.932279 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:53.926764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59594 (local address 127.10.33.124:41865)
0927 15:18:53.926967 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.926974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.927300 (+ 326us) server_negotiation.cc:407] Connection header received
0927 15:18:53.927492 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.927496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.927655 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.927768 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:53.928156 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.928858 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.930211 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.930604 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.930711 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:53.931196 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:53.931717 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:53.931854 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:53.931935 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 15:18:53.932015 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:18:53.946466 1062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:53.937982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42590 (local address 127.10.33.126:44327)
0927 15:18:53.939518 (+ 1536us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.939525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.939547 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:18:53.939686 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.939693 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.939950 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.940155 (+ 205us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:53.940818 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.941937 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.943833 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.944409 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.944534 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:53.945057 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:53.945552 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:53.945669 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:53.946140 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 15:18:53.946235 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1393,"thread_start_us":1261,"threads_started":1}
I20260927 15:18:53.948096 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:53.941205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59602 (local address 127.10.33.124:41865)
0927 15:18:53.941400 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.941404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.941637 (+ 233us) server_negotiation.cc:407] Connection header received
0927 15:18:53.941780 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.941785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.941913 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.942011 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:53.942838 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.943897 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.945828 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.946537 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.946665 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:53.947407 (+ 742us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:53.947507 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:53.947554 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:53.947763 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 15:18:53.947845 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 15:18:53.948120 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:53.938150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32958 (local address 127.10.33.125:43541)
0927 15:18:53.938359 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:53.938366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:53.940930 (+ 2564us) server_negotiation.cc:407] Connection header received
0927 15:18:53.941569 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:53.941575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:53.941668 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:53.941749 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:53.942893 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.943964 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.945954 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:53.946595 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:53.946709 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:53.947384 (+ 675us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:53.947519 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:53.947612 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:53.947777 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 15:18:53.947860 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:18:53.951972 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 673
I20260927 15:18:53.953116 735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:54.065814 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 673
I20260927 15:18:54.082854 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 747
I20260927 15:18:54.083897 810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:54.191411 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 747
I20260927 15:18:54.207809 10373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQegxT3/build/asan/bin/kudu pid 822
I20260927 15:18:54.209030 884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 15:18:54.316799 10373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQegxT3/build/asan/bin/kudu with pid 822
2026-09-27T15:18:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4379 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9406 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (365885 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST